+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210512/A0e --output-run DEEP/20210512/A0e/science#step0/20240315T121610Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 6.0646 seconds; current memory usage: 0.184 Gibyte, delta: 0.043 Gibyte, peak delta: 0.055 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.196 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0052 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0082 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/pipeline_orig.qgraph DEEP/20210512/A0e/science#step0/20240315T121610Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 3.5468 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 9.7556 seconds; current memory usage: 0.287 Gibyte, delta: 0.103 Gibyte, peak delta: 0.091 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.287 Gibyte (main), 0.278 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0298 seconds; current memory usage: 0.287 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.287 Gibyte (main), 0.278 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1054 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210512_A0e_science#step0_20240315T121610Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.2292 seconds; current memory usage: 0.287 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.287 Gibyte (main), 0.278 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1056 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.4224 seconds; current memory usage: 0.289 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.289 Gibyte (main), 0.278 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 32acf5a3-27f2-4180-b2fc-bb5fc4872543 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-15 12:16:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-15 12:16:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 12:16:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-15 12:16:44 proc_lsst.multi:170 [INFO] [multi] job_id 91474 proc_lsst.multi INFO: [multi] job_id 91474 2024-03-15 12:16:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:44 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 91474 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 91474 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 91474 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash ab11f7f2da7efbbacf8ce1e9374de4e4 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/22c637b0-4093-4ab1-aa1f-7a2a9a84d26e_isrForCrosstalkSources_996711_9.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/22c637b0-4093-4ab1-aa1f-7a2a9a84d26e_isrForCrosstalkSources_996711_9.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 25a6bd3a7907fba404790c0c38f42fa2 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/42ca1f15-b6e3-4967-b80b-823aebb0c4b6_isrForCrosstalkSources_996712_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/42ca1f15-b6e3-4967-b80b-823aebb0c4b6_isrForCrosstalkSources_996712_51.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 65c1bfc9129d262747700b49e29cda2f parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/785c6e92-51f6-414b-a889-cdd0db11958e_isrForCrosstalkSources_996709_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/785c6e92-51f6-414b-a889-cdd0db11958e_isrForCrosstalkSources_996709_45.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 3857c52296f212c71fdc30a7bf3d6d0e parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a923413f-f553-4d0a-97dc-98b60cd4478a_isrForCrosstalkSources_996708_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a923413f-f553-4d0a-97dc-98b60cd4478a_isrForCrosstalkSources_996708_35.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 3c46b853c386f8b57ebb27e5de562685 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/60e4f269-548c-4410-8e45-4ad10213276f_isrForCrosstalkSources_996719_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/60e4f269-548c-4410-8e45-4ad10213276f_isrForCrosstalkSources_996719_28.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash c798c61e328e466fdbcbef794ff003b6 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/a8640b44-6478-49ff-b59e-8e25293847e5_isrForCrosstalkSources_996718_49.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/a8640b44-6478-49ff-b59e-8e25293847e5_isrForCrosstalkSources_996718_49.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 228861719e26daec27ba4b6958640425 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b67a555d-4e06-4375-ac9f-729e07ac942e_isrForCrosstalkSources_996705_21.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b67a555d-4e06-4375-ac9f-729e07ac942e_isrForCrosstalkSources_996705_21.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 23e9a8feaf3855a98cd35c0a4826e5e5 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/20581659-8f15-48ae-9182-abe711a5128d_isrForCrosstalkSources_996711_58.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/20581659-8f15-48ae-9182-abe711a5128d_isrForCrosstalkSources_996711_58.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash d2552b9fa08bce76c5c9086e640989a4 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/40371183-530d-4b77-8c85-b4afacc2e445_isrForCrosstalkSources_996720_57.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/40371183-530d-4b77-8c85-b4afacc2e445_isrForCrosstalkSources_996720_57.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 238b9ff4f21d4763061c1daa2650bce3 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6a9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/bb54f20c-4ade-41e6-8d70-5fc4194430b0_isrForCrosstalkSources_996708_45.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/bb54f20c-4ade-41e6-8d70-5fc4194430b0_isrForCrosstalkSources_996708_45.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 97e55ac1a5dc75651d66835e815fe7ab parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8c8dd0c2-dd79-404f-b163-a9ed7d3fd579_isrForCrosstalkSources_996720_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8c8dd0c2-dd79-404f-b163-a9ed7d3fd579_isrForCrosstalkSources_996720_2.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 3d577818519ba34e5657c482fb3f84d5 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/fd9a92dc-acd4-4d13-a3c8-45790338273e_isrForCrosstalkSources_996712_57.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/fd9a92dc-acd4-4d13-a3c8-45790338273e_isrForCrosstalkSources_996712_57.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash cfcdc49d7b1c123c62255e405d37c94f parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/febf64d0-619f-49f7-92e7-6b24a97c601f_isrForCrosstalkSources_996719_18.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/febf64d0-619f-49f7-92e7-6b24a97c601f_isrForCrosstalkSources_996719_18.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash ea7323d872eb3ffa7e50459c9d15a8bf parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a475474c-d189-452c-ba1e-5473e2c55e8b_isrForCrosstalkSources_996708_57.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a475474c-d189-452c-ba1e-5473e2c55e8b_isrForCrosstalkSources_996708_57.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 85aafd8b08da3bbeeae033105a90803e parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/1dc1c8d3-282f-4d9f-918b-99c26eacc31b_isrForCrosstalkSources_996715_8.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/1dc1c8d3-282f-4d9f-918b-99c26eacc31b_isrForCrosstalkSources_996715_8.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash ecb368be92f6e9e7f98877380c321d50 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e6febafe-78b3-4fa1-8299-42b75a60b2db_isrForCrosstalkSources_996706_11.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e6febafe-78b3-4fa1-8299-42b75a60b2db_isrForCrosstalkSources_996706_11.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 054bf1a9bf4c3bfb6b88e8b7b635b524 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/59a20509-e73c-43d8-89c5-0ebc795ca147_isrForCrosstalkSources_996706_12.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/59a20509-e73c-43d8-89c5-0ebc795ca147_isrForCrosstalkSources_996706_12.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 8341475b7232a95aed56d6990271e9fd parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1485fe1b-ebc9-41b7-90ff-2e56a3024261_isrForCrosstalkSources_996717_60.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1485fe1b-ebc9-41b7-90ff-2e56a3024261_isrForCrosstalkSources_996717_60.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash ba3f4bc13654f2a92725e64b579681ed parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8b3831c5-f98e-402c-bc97-0df0f3ac4e9a_isrForCrosstalkSources_996712_9.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8b3831c5-f98e-402c-bc97-0df0f3ac4e9a_isrForCrosstalkSources_996712_9.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash cffb766a6a55635bbc2d4caa89324df4 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/09f557f9-3106-4cc9-930f-9285432821d6_isrForCrosstalkSources_996718_15.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/09f557f9-3106-4cc9-930f-9285432821d6_isrForCrosstalkSources_996718_15.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 7e9774bb115c883bbf9123cc2d712532 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/fc7bd063-b5ce-4455-8884-a5f3d599c53c_isrForCrosstalkSources_996711_17.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/fc7bd063-b5ce-4455-8884-a5f3d599c53c_isrForCrosstalkSources_996711_17.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 67bef6f2a8f9aa0c48923bb68ee9c388 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/88a30682-77e9-42b0-964f-2b2bd0a6ea82_isrForCrosstalkSources_996707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/88a30682-77e9-42b0-964f-2b2bd0a6ea82_isrForCrosstalkSources_996707_59.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 599b75b8263f3e24ee2ed768ae27e203 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c767a7e2-e490-48d2-9895-6ceeeb517165_isrForCrosstalkSources_996709_61.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c767a7e2-e490-48d2-9895-6ceeeb517165_isrForCrosstalkSources_996709_61.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 86a6d2c00772a3338cc552585797d1c9 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2e76b13c-f556-4ba9-9346-17b55ec2fad4_isrForCrosstalkSources_996710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2e76b13c-f556-4ba9-9346-17b55ec2fad4_isrForCrosstalkSources_996710_46.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 81cc9c1ae5cc89b160cabd8f6b1901e7 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/38fc74a6-5303-4196-8cc0-eb34ad22fb6b_isrForCrosstalkSources_996716_36.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/38fc74a6-5303-4196-8cc0-eb34ad22fb6b_isrForCrosstalkSources_996716_36.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash e0757124bef9b1d82674d65568047b98 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/a3a70161-e716-47a2-bb7a-23f8d9c5caef_isrForCrosstalkSources_996718_20.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/a3a70161-e716-47a2-bb7a-23f8d9c5caef_isrForCrosstalkSources_996718_20.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash b9439381096e552cf488b4e90a010616 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ec927082-69e6-4b23-91cc-a21f479556d5_isrForCrosstalkSources_996713_11.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ec927082-69e6-4b23-91cc-a21f479556d5_isrForCrosstalkSources_996713_11.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 967bf5730ce3df38d976267f645fb323 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7c71db36-1ebe-4984-a1db-3c5ad3b62bf7_isrForCrosstalkSources_996712_23.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7c71db36-1ebe-4984-a1db-3c5ad3b62bf7_isrForCrosstalkSources_996712_23.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 868bd0e1f6581694a51e3edfe66764d9 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5a240a42-0fd2-4d55-9d62-b19465144d78_isrForCrosstalkSources_996707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5a240a42-0fd2-4d55-9d62-b19465144d78_isrForCrosstalkSources_996707_48.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 22de3910ecc45ded504311a84e9c11e0 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8ab25203-301e-409c-a314-c84dc0ccbb34_isrForCrosstalkSources_996720_28.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8ab25203-301e-409c-a314-c84dc0ccbb34_isrForCrosstalkSources_996720_28.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash b371c3702226430405c4548e8a3528c5 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/03a7886a-09d1-4b97-94cb-6878b1b6537f_isrForCrosstalkSources_996713_38.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/03a7886a-09d1-4b97-94cb-6878b1b6537f_isrForCrosstalkSources_996713_38.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 2752735237b8eee238e8fc9bd4020bb9 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f1bba586-e963-4c07-85c5-b445652ae7ca_isrForCrosstalkSources_996717_2.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f1bba586-e963-4c07-85c5-b445652ae7ca_isrForCrosstalkSources_996717_2.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash dc202907308913389c42dc1f595c81b7 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/864a25b7-d53e-4040-b646-cd2afece6e1c_isrForCrosstalkSources_996711_4.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/864a25b7-d53e-4040-b646-cd2afece6e1c_isrForCrosstalkSources_996711_4.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 6180900d07d5b81ad434a615c09d98f9 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0bf27885-92a0-43b5-9d56-312b2ae5ba95_isrForCrosstalkSources_996715_53.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0bf27885-92a0-43b5-9d56-312b2ae5ba95_isrForCrosstalkSources_996715_53.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 8d3b489a3f39d9885eedd16df30eb332 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/34ec7e42-97b5-41cd-8602-289f94f2f1cd_isrForCrosstalkSources_996710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/34ec7e42-97b5-41cd-8602-289f94f2f1cd_isrForCrosstalkSources_996710_54.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 06a9414cf848f518af7b354f63f8cfdf parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e82e41f6-0aaf-4515-aa68-f6004f80e760_isrForCrosstalkSources_996715_14.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e82e41f6-0aaf-4515-aa68-f6004f80e760_isrForCrosstalkSources_996715_14.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash ff91e3e98195b1bffb1e562be81d8d0d parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a05be6ae-3d0e-403c-a294-106c7edb763a_isrForCrosstalkSources_996709_6.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a05be6ae-3d0e-403c-a294-106c7edb763a_isrForCrosstalkSources_996709_6.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash add675783ad690103f0294b5d2326e2f parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fb89f2c1-2204-47e4-8d86-c7086b6a58e6_isrForCrosstalkSources_996721_44.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fb89f2c1-2204-47e4-8d86-c7086b6a58e6_isrForCrosstalkSources_996721_44.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 17aae9bcb56e9850bbd9c2055e0bf23a parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6abf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/820f2e1d-d35e-4f6e-8aae-3d5c4d4f94b4_isrForCrosstalkSources_996708_41.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/820f2e1d-d35e-4f6e-8aae-3d5c4d4f94b4_isrForCrosstalkSources_996708_41.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 68ebe68c563e9fbc824892bc697094ae parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6abe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1474773d-fd6e-4279-ba3c-f9fccfa31c75_isrForCrosstalkSources_996721_56.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1474773d-fd6e-4279-ba3c-f9fccfa31c75_isrForCrosstalkSources_996721_56.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash f3bcbee7a3e6b1ee3cea0a72e094cccf parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6abd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c8e38166-f0b0-4be6-a5fb-956186b1b8e7_isrForCrosstalkSources_996709_2.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c8e38166-f0b0-4be6-a5fb-956186b1b8e7_isrForCrosstalkSources_996709_2.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 3f9a91e9fe2c9fb664c55995a148861f parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6abec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/2a42eb7d-2b38-49e2-bb1b-7bc7fbd8892e_isrForCrosstalkSources_996708_56.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/2a42eb7d-2b38-49e2-bb1b-7bc7fbd8892e_isrForCrosstalkSources_996708_56.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash f2210435886f0b785277502efe8f1bee parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6abce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/671059d6-d4dc-4a60-bc00-dde4bcb17531_isrForCrosstalkSources_996717_18.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/671059d6-d4dc-4a60-bc00-dde4bcb17531_isrForCrosstalkSources_996717_18.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 449d0e9a787a1adc88d6553fbf984ddd parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6abc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c67d0c4b-7848-4186-866d-6948be12cbec_isrForCrosstalkSources_996713_17.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c67d0c4b-7848-4186-866d-6948be12cbec_isrForCrosstalkSources_996713_17.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 7862396dc93c92b7e007be0d65eefe06 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6abba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/59d9e6e2-5d1c-4fde-8729-c49f7c6f314c_isrForCrosstalkSources_996720_15.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/59d9e6e2-5d1c-4fde-8729-c49f7c6f314c_isrForCrosstalkSources_996720_15.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash ca7dc4a6e8e42a242fdc14f851148364 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/7c86ce70-adce-4382-9082-0ecc42e02c26_isrForCrosstalkSources_996717_37.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/7c86ce70-adce-4382-9082-0ecc42e02c26_isrForCrosstalkSources_996717_37.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 63b17e066c1a54221fcb0d76489712c1 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6abb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5e078cc3-823b-4903-84b6-967e1ef95827_isrForCrosstalkSources_996719_43.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5e078cc3-823b-4903-84b6-967e1ef95827_isrForCrosstalkSources_996719_43.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 9e7b99b6472290b4d24bbdfa122a7721 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/06b03eec-8a0a-45e2-bcaf-8f0779f93943_isrForCrosstalkSources_996717_6.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/06b03eec-8a0a-45e2-bcaf-8f0779f93943_isrForCrosstalkSources_996717_6.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 6292d9fd492407841d394a0e115d78dd parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f49d01df-024f-48ea-8818-755182c5f8cb_isrForCrosstalkSources_996712_17.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f49d01df-024f-48ea-8818-755182c5f8cb_isrForCrosstalkSources_996712_17.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 75b4c8eb25f8540f87f651784fed78f0 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/aa8ddcfe-c93f-404a-a7c0-3d21621a3dde_isrForCrosstalkSources_996710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/aa8ddcfe-c93f-404a-a7c0-3d21621a3dde_isrForCrosstalkSources_996710_19.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 8c895cd4b25f1f83d6e37f4bd752ef13 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/00a7d73d-ecfd-4fad-8b48-f030395abaf2_isrForCrosstalkSources_996715_29.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/00a7d73d-ecfd-4fad-8b48-f030395abaf2_isrForCrosstalkSources_996715_29.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 1d223a63bc9faa1186548dbfa2a72573 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/398ce131-9e84-42f9-a75c-78c36e264884_isrForCrosstalkSources_996707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/398ce131-9e84-42f9-a75c-78c36e264884_isrForCrosstalkSources_996707_2.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 8c8c4f98285899a2fcbd50a19e64fb7a parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/4fbc94d2-e6f6-4ba2-83fc-95695ec2f334_isrForCrosstalkSources_996709_40.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/4fbc94d2-e6f6-4ba2-83fc-95695ec2f334_isrForCrosstalkSources_996709_40.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash ed6f831036e5af3e0e534469e65cb1bf parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/33bfa71b-c976-4cd9-8767-8297ae874987_isrForCrosstalkSources_996721_19.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/33bfa71b-c976-4cd9-8767-8297ae874987_isrForCrosstalkSources_996721_19.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 5b00471d568c62a78f75f832abe79e28 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6aa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/44324103-6c16-4aeb-bd58-ac85571ff8f3_isrForCrosstalkSources_996721_61.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/44324103-6c16-4aeb-bd58-ac85571ff8f3_isrForCrosstalkSources_996721_61.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 4b80b39ca7b0f6809a66a139e633e7b7 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/7363aeec-2d50-447d-956f-8af167cc345a_isrForCrosstalkSources_996707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/7363aeec-2d50-447d-956f-8af167cc345a_isrForCrosstalkSources_996707_34.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 2f518c4939f0a68bd48700a315ca28c6 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b46a184a-80df-427e-81d0-2dacb2113c43_isrForCrosstalkSources_996715_17.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b46a184a-80df-427e-81d0-2dacb2113c43_isrForCrosstalkSources_996715_17.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash cc99fcd8206f92bfc8a7458a7b932ca9 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f8c46768-a67c-44a1-896f-ec2a59136319_isrForCrosstalkSources_996714_57.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f8c46768-a67c-44a1-896f-ec2a59136319_isrForCrosstalkSources_996714_57.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 86f813e4633ff3568f3ab8827756d4db parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7202c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/be7242a8-5d5b-4740-a1f2-1e34f9cd0632_isrForCrosstalkSources_996711_26.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/be7242a8-5d5b-4740-a1f2-1e34f9cd0632_isrForCrosstalkSources_996711_26.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash adb0be5a9fc083f84d607f1cc1f89e52 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/95469df3-3b50-4abe-8a11-465438a7824f_isrForCrosstalkSources_996713_18.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/95469df3-3b50-4abe-8a11-465438a7824f_isrForCrosstalkSources_996713_18.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 09fdd6e6c8b90fce8c117584069ae3d8 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cbc19148-dece-4155-990a-bfc23c9b858a_isrForCrosstalkSources_996714_23.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cbc19148-dece-4155-990a-bfc23c9b858a_isrForCrosstalkSources_996714_23.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash e8bb7a886340246a71c53bc8bbc98cd1 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/073ff42b-9f55-43e9-b178-cae77a06591b_isrForCrosstalkSources_996710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/073ff42b-9f55-43e9-b178-cae77a06591b_isrForCrosstalkSources_996710_3.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 28f6e23fe3220572cbc7f7ea10518ef3 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/df220403-c22c-4046-8c3e-52d16d0545fd_isrForCrosstalkSources_996707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/df220403-c22c-4046-8c3e-52d16d0545fd_isrForCrosstalkSources_996707_13.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 6fc2372c00b78961fa8efbd619fef3ac parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7205e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/9f8edd92-987a-426d-882e-49e46000dcd7_isrForCrosstalkSources_996712_40.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/9f8edd92-987a-426d-882e-49e46000dcd7_isrForCrosstalkSources_996712_40.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash afc304e1ccaa4f3b692671501cbef46f parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/5b976ab3-0742-46eb-bef7-ca57c55020fc_isrForCrosstalkSources_996714_53.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/5b976ab3-0742-46eb-bef7-ca57c55020fc_isrForCrosstalkSources_996714_53.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 065084dbee8bb7d51e0b273c01a7f3ee parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/eb19f250-f8fa-444b-9ad4-3f82498aa5d9_isrForCrosstalkSources_996716_45.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/eb19f250-f8fa-444b-9ad4-3f82498aa5d9_isrForCrosstalkSources_996716_45.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash a6bceaa03dcf85a48dbe72c3323ab1b7 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7207c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d861bed1-8631-4184-bb2b-3e135ca9fc16_isrForCrosstalkSources_996705_20.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d861bed1-8631-4184-bb2b-3e135ca9fc16_isrForCrosstalkSources_996705_20.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 8558d8d0f45d823f5d8ea8bc876fcd74 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5a0e0fed-bbd1-4c70-80da-bbcfa75c54d9_isrForCrosstalkSources_996721_58.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5a0e0fed-bbd1-4c70-80da-bbcfa75c54d9_isrForCrosstalkSources_996721_58.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 6d860cf4dea43cf8b829558127a8eb6e parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a5e0f51f-0681-4811-9e76-0e787a0f67bd_isrForCrosstalkSources_996708_27.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a5e0f51f-0681-4811-9e76-0e787a0f67bd_isrForCrosstalkSources_996708_27.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 87a993310dd56bdf40755e4a178d7f18 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/14b046d2-e760-4942-89a8-bf79ca54065c_isrForCrosstalkSources_996709_57.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/14b046d2-e760-4942-89a8-bf79ca54065c_isrForCrosstalkSources_996709_57.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 9a383aa6ae71d41779a1cc1af2ab6b8d parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7209a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/445eeb27-3d1d-4b90-8c14-f86584e6ebe0_isrForCrosstalkSources_996719_51.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/445eeb27-3d1d-4b90-8c14-f86584e6ebe0_isrForCrosstalkSources_996719_51.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 11f8aaecf3a7e8975a9dec188b9c89b1 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c23cf8bd-98bd-4509-9516-ee28360ae7fa_isrForCrosstalkSources_996719_49.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c23cf8bd-98bd-4509-9516-ee28360ae7fa_isrForCrosstalkSources_996719_49.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash a07d2ccff0ede33bab699bf85851ca07 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/da3bf7ba-176b-4e6e-8090-afacf281a4c9_isrForCrosstalkSources_996715_48.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/da3bf7ba-176b-4e6e-8090-afacf281a4c9_isrForCrosstalkSources_996715_48.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 7528595378b332dbcd95dd19c5fa8afb parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/9b5e459a-0bce-47a9-b5b1-15390d0f22ff_isrForCrosstalkSources_996720_42.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/9b5e459a-0bce-47a9-b5b1-15390d0f22ff_isrForCrosstalkSources_996720_42.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 88cfeeac8f0be3e7de2b7201f8862796 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/37bf3ecc-8b9c-4274-91cd-1c47a3fd7cc7_isrForCrosstalkSources_996708_4.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/37bf3ecc-8b9c-4274-91cd-1c47a3fd7cc7_isrForCrosstalkSources_996708_4.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 72f7739b01aed8d82655a652d06d7c4b parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/494067ae-c56e-4f9c-9c95-5b3db181c384_isrForCrosstalkSources_996708_30.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/494067ae-c56e-4f9c-9c95-5b3db181c384_isrForCrosstalkSources_996708_30.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 3d54aa2e87a98c995c1e7af2cfd33005 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/346d0b78-19b5-4d85-9f0b-ddd97cf5cc42_isrForCrosstalkSources_996719_37.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/346d0b78-19b5-4d85-9f0b-ddd97cf5cc42_isrForCrosstalkSources_996719_37.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 4f1a91aa200ec56cbdc08e968bdf601e parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/f731dd4c-e4cd-4f0d-aac9-df451eaab7d3_isrForCrosstalkSources_996715_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/f731dd4c-e4cd-4f0d-aac9-df451eaab7d3_isrForCrosstalkSources_996715_58.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 1a209da0a3bc4a3a33e28fbd3b4883ff parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1595f95f-1038-4639-9618-bf267c83ceb3_isrForCrosstalkSources_996712_7.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1595f95f-1038-4639-9618-bf267c83ceb3_isrForCrosstalkSources_996712_7.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 8179380bcd7ee910ea9651ce7cb213d0 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/9874d830-704b-4481-a996-3a128a5cbc54_isrForCrosstalkSources_996719_46.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/9874d830-704b-4481-a996-3a128a5cbc54_isrForCrosstalkSources_996719_46.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash c67ad129da569a8d8e7645a0f85fdad6 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd720fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/73964f24-fb0f-4f32-8556-38f64e21fed7_isrForCrosstalkSources_996718_51.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/73964f24-fb0f-4f32-8556-38f64e21fed7_isrForCrosstalkSources_996718_51.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash b6540bb06f01453f6aa66f2cc9fbbf1d parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/603c8710-f0ff-4eae-bc1e-b0d8c33d9aff_isrForCrosstalkSources_996715_33.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/603c8710-f0ff-4eae-bc1e-b0d8c33d9aff_isrForCrosstalkSources_996715_33.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 8c45832c5375f9270d08026d0d068fcb parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/dfebe05f-7c15-4856-8899-168498555da3_isrForCrosstalkSources_996721_43.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/dfebe05f-7c15-4856-8899-168498555da3_isrForCrosstalkSources_996721_43.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 51a76585669a2a7d1404c945b14ff875 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/9b39daf5-9aa9-44c7-adb5-fe4166658a2c_isrForCrosstalkSources_996711_6.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/9b39daf5-9aa9-44c7-adb5-fe4166658a2c_isrForCrosstalkSources_996711_6.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 702bc342fa6240bed5c03fac5bbb77ae parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/26be4ed8-fe6e-492c-ae7a-e1966c28c722_isrForCrosstalkSources_996710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/26be4ed8-fe6e-492c-ae7a-e1966c28c722_isrForCrosstalkSources_996710_26.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash b10192320a0e07866e58ea2df38982ca parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d9473cda-25dd-4845-b55a-391963c1462f_isrForCrosstalkSources_996705_50.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d9473cda-25dd-4845-b55a-391963c1462f_isrForCrosstalkSources_996705_50.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 0741f5719b721ac55507178c2c5f9823 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/3c1b5a31-bc86-4b1f-915c-b0b0d03b2786_isrForCrosstalkSources_996714_3.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/3c1b5a31-bc86-4b1f-915c-b0b0d03b2786_isrForCrosstalkSources_996714_3.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 61b047fe3a930382ea7e5b1cb23a6ac3 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7214e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2e8501f7-d32d-4e20-950b-fdde51ef6784_isrForCrosstalkSources_996718_11.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2e8501f7-d32d-4e20-950b-fdde51ef6784_isrForCrosstalkSources_996718_11.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 729d2081d3f6ffaaf3c77e01d210f1ba parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6df6a1c9-bb99-4e18-a679-b2c9231747b4_isrForCrosstalkSources_996713_42.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6df6a1c9-bb99-4e18-a679-b2c9231747b4_isrForCrosstalkSources_996713_42.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 1c67704aa516056ee74d10e3e9b5c943 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7216c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e51e5685-7ef3-4ae5-8a58-ae57287b292b_isrForCrosstalkSources_996706_13.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e51e5685-7ef3-4ae5-8a58-ae57287b292b_isrForCrosstalkSources_996706_13.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash bcca447f85364ea9990a74b2a248ffdf parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/986c3a69-3787-4295-ad15-89d4674cdbbc_isrForCrosstalkSources_996708_11.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/986c3a69-3787-4295-ad15-89d4674cdbbc_isrForCrosstalkSources_996708_11.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 76f2d7ec7a7c9bca5b47cda6f4057144 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/28704f90-2440-444c-b90e-decd0c78fe29_isrForCrosstalkSources_996714_29.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/28704f90-2440-444c-b90e-decd0c78fe29_isrForCrosstalkSources_996714_29.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash a847f6157da2a7be29bb974f71f3d9e4 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/7c270093-84f0-48de-aca9-fcfb0e7e4187_isrForCrosstalkSources_996709_41.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/7c270093-84f0-48de-aca9-fcfb0e7e4187_isrForCrosstalkSources_996709_41.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 838a671d0546ca7c9b7ba44a9e963c13 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7218a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/d6a5d52f-c503-4d4a-8e14-25c6c4722612_isrForCrosstalkSources_996721_28.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/d6a5d52f-c503-4d4a-8e14-25c6c4722612_isrForCrosstalkSources_996721_28.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash adb00d19cb609536d636399d401cdd5e parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7219e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/641acaba-587f-49be-b1f6-f7af92190a90_isrForCrosstalkSources_996718_17.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/641acaba-587f-49be-b1f6-f7af92190a90_isrForCrosstalkSources_996718_17.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 7c1c9939ca83db69a42ed5f3ae4b8b58 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/c2529fcf-0776-4147-acf2-58d1bcb08633_isrForCrosstalkSources_996711_12.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/c2529fcf-0776-4147-acf2-58d1bcb08633_isrForCrosstalkSources_996711_12.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 287d392c3f89ceaec2869a61bfb5b6c8 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/3151793c-3f33-4ed4-a1be-99a7266b09f2_isrForCrosstalkSources_996713_13.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/3151793c-3f33-4ed4-a1be-99a7266b09f2_isrForCrosstalkSources_996713_13.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash e14783cfa992364bd8c4d52188e0d980 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d4052aea-ea8b-4437-a97c-f3c00189a6b2_isrForCrosstalkSources_996713_29.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d4052aea-ea8b-4437-a97c-f3c00189a6b2_isrForCrosstalkSources_996713_29.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 9cfaf4ac19c3e10fc5fbdda9de5a05bc parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5874f131-6109-42d1-8b5a-0116ec1a1a03_isrForCrosstalkSources_996710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5874f131-6109-42d1-8b5a-0116ec1a1a03_isrForCrosstalkSources_996710_22.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 67515f4a1249057f4e79782fcce145e5 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/500aaaf2-3973-419a-aa06-796590ae93bc_isrForCrosstalkSources_996707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/500aaaf2-3973-419a-aa06-796590ae93bc_isrForCrosstalkSources_996707_62.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 2f4fe188eed1202d663035e42ae982e0 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/373ee6fe-b81b-4038-8774-f506ee83bed2_isrForCrosstalkSources_996721_38.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/373ee6fe-b81b-4038-8774-f506ee83bed2_isrForCrosstalkSources_996721_38.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash fa91cdbda05c45cb904a5deb2bce8d74 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e84f9300-ebd7-46e6-8b15-1bedf6e9993b_isrForCrosstalkSources_996717_38.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e84f9300-ebd7-46e6-8b15-1bedf6e9993b_isrForCrosstalkSources_996717_38.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 7e3fa9d5851e5f8fe38ee9ba228c2b52 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/cbbc695d-d976-4c7e-85fd-1fecf3bbec82_isrForCrosstalkSources_996713_16.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/cbbc695d-d976-4c7e-85fd-1fecf3bbec82_isrForCrosstalkSources_996713_16.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 6a48ce2443deae86cc6a18779a419383 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/42b0ed02-8bca-4918-b8f8-b8b3d11127d7_isrForCrosstalkSources_996708_60.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/42b0ed02-8bca-4918-b8f8-b8b3d11127d7_isrForCrosstalkSources_996708_60.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 06272fc8eb3c7a915217cca54e94256b parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a6c49934-f172-499a-b534-84adf6f94baf_isrForCrosstalkSources_996706_24.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a6c49934-f172-499a-b534-84adf6f94baf_isrForCrosstalkSources_996706_24.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash b529dd16e935356874dd973887a230c9 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd721f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/d9dac860-9a7f-4f45-b5d6-f0afb5ecee4a_isrForCrosstalkSources_996710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/d9dac860-9a7f-4f45-b5d6-f0afb5ecee4a_isrForCrosstalkSources_996710_2.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 523f1e02327d16b530b1b27ba368a606 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/47a83f36-5d89-4d7c-ae8d-1750a2c91cb7_isrForCrosstalkSources_996705_30.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/47a83f36-5d89-4d7c-ae8d-1750a2c91cb7_isrForCrosstalkSources_996705_30.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 537654709649c7583dc31264da3b31d5 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed905e3c-97de-443a-9538-077c342866a4_isrForCrosstalkSources_996709_22.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed905e3c-97de-443a-9538-077c342866a4_isrForCrosstalkSources_996709_22.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 30994fec4f238ed15522ffb299f9064e parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/e7c0be03-c22d-4808-8bf1-d1c271ee4039_isrForCrosstalkSources_996712_18.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/e7c0be03-c22d-4808-8bf1-d1c271ee4039_isrForCrosstalkSources_996712_18.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 4d7471266af3ca448f41cdf2a9a645e0 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1a2a9ce3-a21e-448d-8dad-8eda983f2445_isrForCrosstalkSources_996707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1a2a9ce3-a21e-448d-8dad-8eda983f2445_isrForCrosstalkSources_996707_30.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 966665c8d44e0ae70480637b133bea2f parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/010fde18-fa2c-4a66-acbe-357ff32a6584_isrForCrosstalkSources_996715_54.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/010fde18-fa2c-4a66-acbe-357ff32a6584_isrForCrosstalkSources_996715_54.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 0943984d8b74351a177463c3f7f64115 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7223e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/fd594b90-b290-4eaa-aa36-09606739e326_isrForCrosstalkSources_996716_61.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/fd594b90-b290-4eaa-aa36-09606739e326_isrForCrosstalkSources_996716_61.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash ec474d3433ea0b3a27b32aa761c3ecd5 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c71cb0b7-d690-4c8d-8408-5686cab8bc1d_isrForCrosstalkSources_996713_53.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c71cb0b7-d690-4c8d-8408-5686cab8bc1d_isrForCrosstalkSources_996713_53.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 6698b16d8beaf0c7f517ae97faf702ae parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/31e88c38-25f1-4d36-bd08-bee26e8f06fb_isrForCrosstalkSources_996719_2.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/31e88c38-25f1-4d36-bd08-bee26e8f06fb_isrForCrosstalkSources_996719_2.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 6352e3bc3ace1816b2d4846535e1ea92 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/19091f73-8361-4c02-bbd5-6e53116b20a3_isrForCrosstalkSources_996718_24.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/19091f73-8361-4c02-bbd5-6e53116b20a3_isrForCrosstalkSources_996718_24.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 4ee23799d49d1fdcbf0af232a11d6b17 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7225c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/11c8df40-52a3-45f3-989c-57dd70c4f91e_isrForCrosstalkSources_996706_1.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/11c8df40-52a3-45f3-989c-57dd70c4f91e_isrForCrosstalkSources_996706_1.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash cd0986d450f0c0d7ee2c962edb845266 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/2aaf304a-c941-4892-a23a-98daeedbcc24_isrForCrosstalkSources_996715_25.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/2aaf304a-c941-4892-a23a-98daeedbcc24_isrForCrosstalkSources_996715_25.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash c0d4f34f327fc67c6ced228921f73248 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7227a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fecadec6-691f-414c-aa7d-efcfbb49562b_isrForCrosstalkSources_996717_29.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fecadec6-691f-414c-aa7d-efcfbb49562b_isrForCrosstalkSources_996717_29.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 462557a352d156148525eaf4962b5dbb parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/0f4ca18b-b748-4685-92fd-3c94275a64a1_isrForCrosstalkSources_996709_17.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/0f4ca18b-b748-4685-92fd-3c94275a64a1_isrForCrosstalkSources_996709_17.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash f19ea136755e7cb48135ecbeb0e4fac8 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/54affc5d-9f3a-4936-9ddf-b0189a9dd708_isrForCrosstalkSources_996705_37.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/54affc5d-9f3a-4936-9ddf-b0189a9dd708_isrForCrosstalkSources_996705_37.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 2cecc81ad070f2905936dc5840f4268e parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d263253b-9a9d-413b-a2fe-77c503713ec6_isrForCrosstalkSources_996711_36.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d263253b-9a9d-413b-a2fe-77c503713ec6_isrForCrosstalkSources_996711_36.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash e1a5dcad7fe5f577778e8707c1325e5d parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/eb4801b1-78d7-460a-bc2a-63248d9a9047_isrForCrosstalkSources_996713_6.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/eb4801b1-78d7-460a-bc2a-63248d9a9047_isrForCrosstalkSources_996713_6.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 8625329a6740b8ab86a2f241b477545b parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/fd6bc526-0331-4eb4-ad42-9cde873b62fa_isrForCrosstalkSources_996718_54.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/fd6bc526-0331-4eb4-ad42-9cde873b62fa_isrForCrosstalkSources_996718_54.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 12d4fd325147ea2200f122936ea36975 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/8d1c2462-4c65-4196-8d85-3668b1230408_isrForCrosstalkSources_996708_8.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/8d1c2462-4c65-4196-8d85-3668b1230408_isrForCrosstalkSources_996708_8.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash e36d4868ce816bfe29ac3ecb6f558a18 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/67121a15-63c4-42b1-bd59-1326b00e8338_isrForCrosstalkSources_996719_30.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/67121a15-63c4-42b1-bd59-1326b00e8338_isrForCrosstalkSources_996719_30.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash bd9a7081154f2f646b8ace7c42215619 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/ff0e2528-f55d-409f-8421-57a93103e3a6_isrForCrosstalkSources_996711_29.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/ff0e2528-f55d-409f-8421-57a93103e3a6_isrForCrosstalkSources_996711_29.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash e6e37118ca9556524bb7a9d78cd34589 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0889d857-60ec-4e67-8963-d109af2105a3_isrForCrosstalkSources_996719_47.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0889d857-60ec-4e67-8963-d109af2105a3_isrForCrosstalkSources_996719_47.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 24af83e4847601241e71c713c5563381 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5b0ab9a5-fde0-45b9-8b64-a68e99768b00_isrForCrosstalkSources_996709_37.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5b0ab9a5-fde0-45b9-8b64-a68e99768b00_isrForCrosstalkSources_996709_37.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash fd9caf9f858b41e05b95251686d18479 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/8f915c1b-0c12-4504-b223-e4cb552c009b_isrForCrosstalkSources_996709_3.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/8f915c1b-0c12-4504-b223-e4cb552c009b_isrForCrosstalkSources_996709_3.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 730a946808127e8fdedcd8df93f2b403 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/629f453c-b418-4449-9813-a5c1b387367f_isrForCrosstalkSources_996717_41.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/629f453c-b418-4449-9813-a5c1b387367f_isrForCrosstalkSources_996717_41.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 59e9049291e79610ce4bd13155027d63 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd722f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/b73a9de8-989c-4fbf-973c-e3e0bfee22ba_isrForCrosstalkSources_996711_23.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/b73a9de8-989c-4fbf-973c-e3e0bfee22ba_isrForCrosstalkSources_996711_23.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash aed2dee1f61fc0dd3c8e5e03a11551e5 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/b6bc0f3c-a75e-49b7-9b73-759919150afa_isrForCrosstalkSources_996719_6.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/b6bc0f3c-a75e-49b7-9b73-759919150afa_isrForCrosstalkSources_996719_6.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 2f553809baa8f2c9a5683982ee13fa1d parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f55f0f18-e923-479f-978e-cc179ac8bba8_isrForCrosstalkSources_996717_17.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f55f0f18-e923-479f-978e-cc179ac8bba8_isrForCrosstalkSources_996717_17.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash d92a6b1e597534970680ad3c8768f9f3 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/697bc477-6d7f-41dd-b4b5-67a56945f1e1_isrForCrosstalkSources_996707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/697bc477-6d7f-41dd-b4b5-67a56945f1e1_isrForCrosstalkSources_996707_53.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash b84fe8c4ba00f914fe3f1cf4f33ff2cb parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7232e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/a76d428d-4bff-409b-8ad4-837ea90cabe6_isrForCrosstalkSources_996710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/a76d428d-4bff-409b-8ad4-837ea90cabe6_isrForCrosstalkSources_996710_23.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 177993b787293585543f9cbf5ebb0654 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5a91691b-8be4-41d3-b399-d017849df239_isrForCrosstalkSources_996721_34.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5a91691b-8be4-41d3-b399-d017849df239_isrForCrosstalkSources_996721_34.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 7d669faf718b6758804952ebbb43baef parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:45 proc_lsst.multi:146 [INFO] found job 91474 in provider local proc_lsst.multi INFO: found job 91474 in provider local parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/264f0532-8166-41f3-a04d-e519148cb946_isrForCrosstalkSources_996714_40.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/264f0532-8166-41f3-a04d-e519148cb946_isrForCrosstalkSources_996714_40.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 3fe894959d09a4cc388533fdc74ce216 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/63d18a8a-adfc-4005-b737-277a289f60b1_isrForCrosstalkSources_996716_6.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/63d18a8a-adfc-4005-b737-277a289f60b1_isrForCrosstalkSources_996716_6.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 8778c57e66e0aaa84fced303639ad713 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7234c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/eecab347-4729-49fb-bda5-825e76a358de_isrForCrosstalkSources_996718_36.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/eecab347-4729-49fb-bda5-825e76a358de_isrForCrosstalkSources_996718_36.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 95a86b372d7361d84f87d033f1f57d9d parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7200e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d0f408aa-1721-46e0-a385-1edf379529be_isrForCrosstalkSources_996705_41.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d0f408aa-1721-46e0-a385-1edf379529be_isrForCrosstalkSources_996705_41.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 25abc942ecf0408d24d2cb304ab2e581 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7237e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 140 parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a8c4e0fa-5b92-4ce1-aa6b-fa1f951911c1_isrForCrosstalkSources_996706_29.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a8c4e0fa-5b92-4ce1-aa6b-fa1f951911c1_isrForCrosstalkSources_996706_29.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash f3e597d79323fe123ba855eb32505911 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Executor multi has 140 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 34 more blocks parsl.executors.status_handling INFO: Scaling out by 34 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-15 12:16:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-15 12:16:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 12:16:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/34ed4139-e813-489a-a8c2-1704c12cfc19_isrForCrosstalkSources_996718_35.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/34ed4139-e813-489a-a8c2-1704c12cfc19_isrForCrosstalkSources_996718_35.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 121e6a537a978e8cffc2c44fbd5ca81a parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7239c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e8f17f8e-d006-4044-90dc-a0a9f73e769d_isrForCrosstalkSources_996716_62.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e8f17f8e-d006-4044-90dc-a0a9f73e769d_isrForCrosstalkSources_996716_62.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 2a664ca03b5bc4ab2fb00246523ac1ac parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/846beb61-dd43-4b2f-99d1-24825b436769_isrForCrosstalkSources_996715_5.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/846beb61-dd43-4b2f-99d1-24825b436769_isrForCrosstalkSources_996715_5.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 4bdff9b5546af2605f7badf53b7d215c parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/abb425bd-c715-472d-9b16-64e9e3d2d771_isrForCrosstalkSources_996721_51.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/abb425bd-c715-472d-9b16-64e9e3d2d771_isrForCrosstalkSources_996721_51.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash ffbaf59bae1e13331c812f3fa349c2c1 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/95c4c788-ee8a-4834-8f5e-2c1c80eaef42_isrForCrosstalkSources_996720_7.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/95c4c788-ee8a-4834-8f5e-2c1c80eaef42_isrForCrosstalkSources_996720_7.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash efe86b46bc82d9f8013148738f6bf0ae parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd6ab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/10a4716d-bffe-4b19-9d12-1de0f357603a_isrForCrosstalkSources_996706_34.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/10a4716d-bffe-4b19-9d12-1de0f357603a_isrForCrosstalkSources_996706_34.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash b5f2a604dbe1df2dfa8877388b73d8d7 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/87799243-9d84-435c-9f07-cbbec3490e08_isrForCrosstalkSources_996717_53.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/87799243-9d84-435c-9f07-cbbec3490e08_isrForCrosstalkSources_996717_53.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 3dd48ebc88fe15b0f32c61149c9d02aa parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f44390b7-bfca-4a26-aebe-914a180a3f9d_isrForCrosstalkSources_996707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f44390b7-bfca-4a26-aebe-914a180a3f9d_isrForCrosstalkSources_996707_58.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash b91dd1837e5231dea0af6715c9bcad2d parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2106dbe2-2503-441c-a6ad-44e72d53b023_isrForCrosstalkSources_996712_14.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2106dbe2-2503-441c-a6ad-44e72d53b023_isrForCrosstalkSources_996712_14.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 46ab80783073883fcfe3bab9b44d0b8b parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4221f3b4-fac6-4909-b843-0d9f1c9e40b3_isrForCrosstalkSources_996719_27.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4221f3b4-fac6-4909-b843-0d9f1c9e40b3_isrForCrosstalkSources_996719_27.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash e1ba8115841858a28a2378e8c976aaa8 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c58ec095-afee-4bfc-85dd-712b731038fc_isrForCrosstalkSources_996709_60.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c58ec095-afee-4bfc-85dd-712b731038fc_isrForCrosstalkSources_996709_60.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash da2d4ef067d5643a179bdafbe02d3046 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c7a8a7d3-5233-4a80-8303-f0930f10dcde_isrForCrosstalkSources_996706_18.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c7a8a7d3-5233-4a80-8303-f0930f10dcde_isrForCrosstalkSources_996706_18.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash a0ed7674717ebb2d8493cedec5fdb4b2 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ee4c960b-a5b1-42e9-81a4-f4002f2e5193_isrForCrosstalkSources_996707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ee4c960b-a5b1-42e9-81a4-f4002f2e5193_isrForCrosstalkSources_996707_10.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 210a2b8322802947a77c8fff569037b9 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0c4a3c5c-813d-4f64-9433-4414bffc9df4_isrForCrosstalkSources_996719_59.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0c4a3c5c-813d-4f64-9433-4414bffc9df4_isrForCrosstalkSources_996719_59.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 9103b738167830b722ba1e795553aa1f parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/268acd7b-c651-426a-bd1b-e4f17d5cc5bd_isrForCrosstalkSources_996719_9.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/268acd7b-c651-426a-bd1b-e4f17d5cc5bd_isrForCrosstalkSources_996719_9.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash c21ad54cc03dea1c197fa75bac09ae4b parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/1319832e-e96b-4a80-8a46-b648de45dd13_isrForCrosstalkSources_996710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/1319832e-e96b-4a80-8a46-b648de45dd13_isrForCrosstalkSources_996710_39.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 7cc378786c9bac146c7d99938ce97b49 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/dbd72520-797e-4b67-a1c0-bf57e17b9daa_isrForCrosstalkSources_996705_15.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/dbd72520-797e-4b67-a1c0-bf57e17b9daa_isrForCrosstalkSources_996705_15.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 255f82d689b5ef9930f8435e160fa46b parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c15991ac-f2c2-4125-9422-41e725deb7c3_isrForCrosstalkSources_996708_51.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c15991ac-f2c2-4125-9422-41e725deb7c3_isrForCrosstalkSources_996708_51.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash bdc5f510c1a0d23f506fc80ed7d05637 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7cdf0e33-7889-42c8-87ae-59eb1b9a4abd_isrForCrosstalkSources_996721_49.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7cdf0e33-7889-42c8-87ae-59eb1b9a4abd_isrForCrosstalkSources_996721_49.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash f88e6a99bf1f25529b0a3f66559541e5 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5d15fd97-b580-483e-b375-917dae2b2715_isrForCrosstalkSources_996720_49.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5d15fd97-b580-483e-b375-917dae2b2715_isrForCrosstalkSources_996720_49.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash be686c11ceaff443a3547c575759a928 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/28d9893a-c886-4264-b86b-0e0e117abdf3_isrForCrosstalkSources_996710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/28d9893a-c886-4264-b86b-0e0e117abdf3_isrForCrosstalkSources_996710_18.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash c88a0ee80d64e7642825fdbd3ee0c8a1 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/941fb006-20de-4369-9d70-18a7df1db5b3_isrForCrosstalkSources_996721_14.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/941fb006-20de-4369-9d70-18a7df1db5b3_isrForCrosstalkSources_996721_14.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 2a21af7eddfdb5135a81eb5210277ae0 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/05d6ead2-9a8b-41fc-9b6b-2851d76df13f_isrForCrosstalkSources_996712_12.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/05d6ead2-9a8b-41fc-9b6b-2851d76df13f_isrForCrosstalkSources_996712_12.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash f85d9f210a003657a31634c76b242866 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/0cc3851e-c2b0-4d0f-8cc5-f29db727d8a9_isrForCrosstalkSources_996708_47.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/0cc3851e-c2b0-4d0f-8cc5-f29db727d8a9_isrForCrosstalkSources_996708_47.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash c8c241aec0f039d3de27bc120b88da50 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/f56bcb0b-91f1-40a9-b588-fffebd90fb9e_isrForCrosstalkSources_996720_20.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/f56bcb0b-91f1-40a9-b588-fffebd90fb9e_isrForCrosstalkSources_996720_20.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 6ead0b98dacc06807cf2ee143227698a parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f874b410-6a8b-405e-83c6-b975243075f8_isrForCrosstalkSources_996710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f874b410-6a8b-405e-83c6-b975243075f8_isrForCrosstalkSources_996710_56.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash e567747cb5b72c8278b2b323359625a8 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/64013c5a-265f-4846-8175-06da26538ee1_isrForCrosstalkSources_996715_41.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/64013c5a-265f-4846-8175-06da26538ee1_isrForCrosstalkSources_996715_41.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash f57c1883a954098c617e6c5cb7c25ef6 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0787b34c-6017-4b8d-b6bf-406915061cc0_isrForCrosstalkSources_996715_4.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0787b34c-6017-4b8d-b6bf-406915061cc0_isrForCrosstalkSources_996715_4.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 4bbfc833d1510afa34f69aa3575d04e5 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f9188824-f701-4d69-89f3-bb3c2559bdac_isrForCrosstalkSources_996712_39.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f9188824-f701-4d69-89f3-bb3c2559bdac_isrForCrosstalkSources_996712_39.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 24450d197805fedebdd5170594809027 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/94374850-ef22-46d4-8109-5964eaef9a6e_isrForCrosstalkSources_996713_39.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/94374850-ef22-46d4-8109-5964eaef9a6e_isrForCrosstalkSources_996713_39.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 565bf1ed703474abda63ea7ffcbdecc7 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ad4897ab-f4d9-44fb-99d3-17049694d12f_isrForCrosstalkSources_996717_35.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ad4897ab-f4d9-44fb-99d3-17049694d12f_isrForCrosstalkSources_996717_35.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 6f2752031b62880ae111618a915d05bb parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/37bc1498-fbb8-444f-a674-06f1a2892ead_isrForCrosstalkSources_996719_13.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/37bc1498-fbb8-444f-a674-06f1a2892ead_isrForCrosstalkSources_996719_13.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash b4bd2f55cbe723c9b4a2f2bf0bacaeb0 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/a8b7ab13-6a36-45bc-aa61-9ca5fd35232b_isrForCrosstalkSources_996707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/a8b7ab13-6a36-45bc-aa61-9ca5fd35232b_isrForCrosstalkSources_996707_55.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash bdfe3fd615f13df145cf885da8192574 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e076adc5-fe61-46e6-97f4-7f6a70a981ca_isrForCrosstalkSources_996717_58.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e076adc5-fe61-46e6-97f4-7f6a70a981ca_isrForCrosstalkSources_996717_58.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 3a46cc8d96df7e4a51e9159951739034 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/64af2a51-9de0-42db-a795-f718fbd1f248_isrForCrosstalkSources_996715_20.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/64af2a51-9de0-42db-a795-f718fbd1f248_isrForCrosstalkSources_996715_20.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash b753476021afbbda945465290ceba3be parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/a45309e1-0449-4df1-8055-b3728ca4c18a_isrForCrosstalkSources_996715_60.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/a45309e1-0449-4df1-8055-b3728ca4c18a_isrForCrosstalkSources_996715_60.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash acd74ed55c81ae00bcb7a5ef8d950d71 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/53675ce4-6043-4472-afa6-e91e6250b595_isrForCrosstalkSources_996706_10.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/53675ce4-6043-4472-afa6-e91e6250b595_isrForCrosstalkSources_996706_10.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 428b01ab7be40737edd71bbdac89ac0b parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/46696e26-72df-4d9b-a7a6-ec4902d78c59_isrForCrosstalkSources_996718_59.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/46696e26-72df-4d9b-a7a6-ec4902d78c59_isrForCrosstalkSources_996718_59.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 0e5785cfbd00cdc9f22b108e3e04e217 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/445ffb51-ba77-4b7a-9086-376848caf563_isrForCrosstalkSources_996713_31.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/445ffb51-ba77-4b7a-9086-376848caf563_isrForCrosstalkSources_996713_31.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 2f21bed746bb62be57ec55613248665b parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d45fa327-4fa6-48ae-a8a5-94ccaa7123a2_isrForCrosstalkSources_996713_58.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d45fa327-4fa6-48ae-a8a5-94ccaa7123a2_isrForCrosstalkSources_996713_58.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash a8c6497e75875425b6b6cfdd983962b3 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/05f59f4c-b6ad-4d39-989d-fc00153ae784_isrForCrosstalkSources_996715_24.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/05f59f4c-b6ad-4d39-989d-fc00153ae784_isrForCrosstalkSources_996715_24.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 6314d8b92dadcc4c75cf3ea08320fbec parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f2850d30-bd1a-4a8c-aba1-d2b687e114b2_isrForCrosstalkSources_996708_5.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f2850d30-bd1a-4a8c-aba1-d2b687e114b2_isrForCrosstalkSources_996708_5.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash a7b26d6b7155432e15305df4a93cbee4 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/8816fabc-73b6-43c1-ab86-4ddb00aa6b26_isrForCrosstalkSources_996716_42.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/8816fabc-73b6-43c1-ab86-4ddb00aa6b26_isrForCrosstalkSources_996716_42.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 39704e3cfcd8c57bb8ffd4d0dfa72046 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/16a1baa6-ca9d-452e-897f-56573a9a27de_isrForCrosstalkSources_996707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/16a1baa6-ca9d-452e-897f-56573a9a27de_isrForCrosstalkSources_996707_4.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 4c7d57259b9beb317b6a006cc4856921 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/ed2d2a3e-8328-45bf-99bb-c973ddb6731f_isrForCrosstalkSources_996721_55.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/ed2d2a3e-8328-45bf-99bb-c973ddb6731f_isrForCrosstalkSources_996721_55.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 4544777f3fd38e8ea42ca8d9860d85bd parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/3407e620-b271-41d1-bd6a-1fd2c1f52abf_isrForCrosstalkSources_996708_52.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/3407e620-b271-41d1-bd6a-1fd2c1f52abf_isrForCrosstalkSources_996708_52.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash b93c0a76f5be884227f0d6815440d631 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e4ef79c5-dbbd-41d3-988b-a80bac02528a_isrForCrosstalkSources_996707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e4ef79c5-dbbd-41d3-988b-a80bac02528a_isrForCrosstalkSources_996707_12.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 960529964654467b8ead5109db86ec9a parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/16835765-60c2-4d12-a600-f854f3f000c6_isrForCrosstalkSources_996712_43.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/16835765-60c2-4d12-a600-f854f3f000c6_isrForCrosstalkSources_996712_43.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 4d65679c64ea4cc2f53bcb4e84098bd0 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed9d94dd-be70-4f39-943f-d9288f6cf75a_isrForCrosstalkSources_996709_18.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed9d94dd-be70-4f39-943f-d9288f6cf75a_isrForCrosstalkSources_996709_18.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 090e1683d69199f8dcbd813932495aba parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/52e40cfe-24a3-4799-8cb6-163c9e2ec483_isrForCrosstalkSources_996716_54.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/52e40cfe-24a3-4799-8cb6-163c9e2ec483_isrForCrosstalkSources_996716_54.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 7af1d9634c484bf603c48af1d3c2b2b6 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/1f76402c-8200-4f7c-857a-7559ccf36d78_isrForCrosstalkSources_996713_4.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/1f76402c-8200-4f7c-857a-7559ccf36d78_isrForCrosstalkSources_996713_4.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash e807e00400010e765d71d02541875cf7 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/20a624d3-9f06-4f3f-91d7-8b786bc9bf63_isrForCrosstalkSources_996711_45.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/20a624d3-9f06-4f3f-91d7-8b786bc9bf63_isrForCrosstalkSources_996711_45.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 3b0769d277b4717b70a674669a67d300 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1a74531c-cf5b-445e-8d33-6206c9417e6e_isrForCrosstalkSources_996717_51.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1a74531c-cf5b-445e-8d33-6206c9417e6e_isrForCrosstalkSources_996717_51.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 38f7afb66aa70dabfb314d6aebef258a parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8434f56a-f870-4839-8c46-3d2bc9069f80_isrForCrosstalkSources_996718_58.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8434f56a-f870-4839-8c46-3d2bc9069f80_isrForCrosstalkSources_996718_58.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 588475a11bbff84ee55fc4e6f1acc480 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/05dab6cf-6307-436d-8e2d-5fde335fde03_isrForCrosstalkSources_996715_22.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/05dab6cf-6307-436d-8e2d-5fde335fde03_isrForCrosstalkSources_996715_22.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash a7c2c5f52b4e6b12ccfc181c23716fae parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a94cd407-7dfc-4c5a-9594-76260b3bf673_isrForCrosstalkSources_996705_57.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a94cd407-7dfc-4c5a-9594-76260b3bf673_isrForCrosstalkSources_996705_57.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 283788ff756729c1babd1265f8fb9184 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/4f7896cc-503a-44f1-8f31-cb78fc5619d3_isrForCrosstalkSources_996715_35.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/4f7896cc-503a-44f1-8f31-cb78fc5619d3_isrForCrosstalkSources_996715_35.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash b63c0052e1011ec4ab7809f2d87b4629 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/210910ce-f1b6-41b2-83c2-36d60a401bd8_isrForCrosstalkSources_996705_6.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/210910ce-f1b6-41b2-83c2-36d60a401bd8_isrForCrosstalkSources_996705_6.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash caee2b1a7e120e03f69f87ea56c12543 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d715ab74-a238-439b-94f2-83a2450eb465_isrForCrosstalkSources_996705_56.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d715ab74-a238-439b-94f2-83a2450eb465_isrForCrosstalkSources_996705_56.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash cbb793d1127296ef6e05db5974ca3ef1 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/b31be672-05a3-4c97-8a2d-535c5cc6ac92_isrForCrosstalkSources_996707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/b31be672-05a3-4c97-8a2d-535c5cc6ac92_isrForCrosstalkSources_996707_19.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash ba99e572815653ca1b25c339bb668d67 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4c7e5425-276e-4083-a7fc-cd7fb6c6154c_isrForCrosstalkSources_996714_54.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4c7e5425-276e-4083-a7fc-cd7fb6c6154c_isrForCrosstalkSources_996714_54.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 657e03b4a349e25e7f0c8e743868b41e parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/86f725d1-4171-42d3-b321-7098ee9ec2c1_isrForCrosstalkSources_996721_1.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/86f725d1-4171-42d3-b321-7098ee9ec2c1_isrForCrosstalkSources_996721_1.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash a6dd24cd14a6317192517e840107d623 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/c4272672-9bb5-43c4-b453-d3bad696fb2c_isrForCrosstalkSources_996710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/c4272672-9bb5-43c4-b453-d3bad696fb2c_isrForCrosstalkSources_996710_6.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 5588da037bc82959e17d88795918a808 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/700f984c-957c-4097-be4d-d3bd5330e4ab_isrForCrosstalkSources_996706_30.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/700f984c-957c-4097-be4d-d3bd5330e4ab_isrForCrosstalkSources_996706_30.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash c58ffd9d95a4c75cb5b378cc7f228119 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/76eee212-b0a6-4dc9-8bcf-c7cd43711b1c_isrForCrosstalkSources_996707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/76eee212-b0a6-4dc9-8bcf-c7cd43711b1c_isrForCrosstalkSources_996707_1.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 6d75b5c3bd4a92df40aa57cc2d106ae7 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1cfb3f4e-311b-4c9d-a292-f5d9cd768494_isrForCrosstalkSources_996705_12.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1cfb3f4e-311b-4c9d-a292-f5d9cd768494_isrForCrosstalkSources_996705_12.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 0874dcaafd0aca8d98f20279608f8a1d parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/3087a346-5828-469b-87cd-9405038f2ca0_isrForCrosstalkSources_996713_12.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/3087a346-5828-469b-87cd-9405038f2ca0_isrForCrosstalkSources_996713_12.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 505d0f562c1eb9b6d7c0e56c48596c78 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/067341eb-b9f2-4f24-befc-7b10a81fa868_isrForCrosstalkSources_996718_5.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/067341eb-b9f2-4f24-befc-7b10a81fa868_isrForCrosstalkSources_996718_5.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 80fdccaf65aa251890b7d3f5a437e238 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/743311d0-23a1-49c7-9a2f-695f5b4c3339_isrForCrosstalkSources_996718_2.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/743311d0-23a1-49c7-9a2f-695f5b4c3339_isrForCrosstalkSources_996718_2.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 4fc250c52774dc9b5be794335b787ce0 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8bf09523-e2d3-4b64-9436-a0086ee6064c_isrForCrosstalkSources_996718_43.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8bf09523-e2d3-4b64-9436-a0086ee6064c_isrForCrosstalkSources_996718_43.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 35c6fab434c2693f45f89316101d8580 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/7a163f6b-a650-43c9-aafa-83c84614ab28_isrForCrosstalkSources_996708_55.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/7a163f6b-a650-43c9-aafa-83c84614ab28_isrForCrosstalkSources_996708_55.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash f342a6f699578f8a3c462dbca63be25b parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/3300418f-9647-4611-b936-4437f99668fd_isrForCrosstalkSources_996717_34.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/3300418f-9647-4611-b936-4437f99668fd_isrForCrosstalkSources_996717_34.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash c360c2e347d7f5889ab871a120aeef29 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ea306c0f-d13f-435b-9dbd-ed3575b8a682_isrForCrosstalkSources_996718_19.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ea306c0f-d13f-435b-9dbd-ed3575b8a682_isrForCrosstalkSources_996718_19.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash e4cc4e208649cd1a5f2fbddb0d3d3889 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ec54d3cf-e705-4645-bb6f-f23762027732_isrForCrosstalkSources_996705_5.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ec54d3cf-e705-4645-bb6f-f23762027732_isrForCrosstalkSources_996705_5.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 8291cf0af0415f51924f13fd362aa64d parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/67b23847-fdf3-4650-be70-844ab0a2eefb_isrForCrosstalkSources_996718_39.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/67b23847-fdf3-4650-be70-844ab0a2eefb_isrForCrosstalkSources_996718_39.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 57c5c06c80ee5deb47a41da6da234f1a parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/df580fd4-f578-422c-a420-36c59b5d1800_isrForCrosstalkSources_996716_10.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/df580fd4-f578-422c-a420-36c59b5d1800_isrForCrosstalkSources_996716_10.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 144f9812e87f1e6845f46b60998c2232 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d2da78b4-60f6-412b-8e86-905663da200b_isrForCrosstalkSources_996709_38.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d2da78b4-60f6-412b-8e86-905663da200b_isrForCrosstalkSources_996709_38.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash aa7b781b4812a99f47af136b515f0518 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/4ea9d24a-13e5-424d-b9c2-44b5407dd71f_isrForCrosstalkSources_996709_30.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/4ea9d24a-13e5-424d-b9c2-44b5407dd71f_isrForCrosstalkSources_996709_30.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 9e3a03d5cd175b75e9f4210e9be6e36f parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6fdb446e-6921-457b-a7f2-3e207bab1da8_isrForCrosstalkSources_996709_33.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6fdb446e-6921-457b-a7f2-3e207bab1da8_isrForCrosstalkSources_996709_33.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 42fe58347f77d61f8742f25b6e340ff3 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd360c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/c51047bb-d4c9-43a8-81ff-5aab7a923c2e_isrForCrosstalkSources_996711_15.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/c51047bb-d4c9-43a8-81ff-5aab7a923c2e_isrForCrosstalkSources_996711_15.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 3ed00693a4fe9ea9f6b0ceff73b86a44 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508dd7ff7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/011c2a6b-22d3-4e7b-be7b-04ec93c7d5c7_isrForCrosstalkSources_996721_17.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/011c2a6b-22d3-4e7b-be7b-04ec93c7d5c7_isrForCrosstalkSources_996721_17.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 48931b3e5c98b68d3507f0a7b42bd6d7 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9a08f871-1b3d-4e21-be1f-9014e24cf072_isrForCrosstalkSources_996714_41.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9a08f871-1b3d-4e21-be1f-9014e24cf072_isrForCrosstalkSources_996714_41.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 988c8c049893d48f2928021a1f46bcff parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/7b5793d9-5e54-4b47-8877-8b4d3abc474a_isrForCrosstalkSources_996705_40.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/7b5793d9-5e54-4b47-8877-8b4d3abc474a_isrForCrosstalkSources_996705_40.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 5f71d0bb77063ee0e7634fe8774bc2af parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/bb79650b-fc1a-4f25-9a36-ea90fa41184d_isrForCrosstalkSources_996715_16.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/bb79650b-fc1a-4f25-9a36-ea90fa41184d_isrForCrosstalkSources_996715_16.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash fe5b07bccc42553f7ba0d34ddece2528 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/433829a4-d315-4e39-a7e6-a6cc4f277fcb_isrForCrosstalkSources_996706_56.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/433829a4-d315-4e39-a7e6-a6cc4f277fcb_isrForCrosstalkSources_996706_56.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 45f419c258dfe2ee3c063122658f8ae0 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/12ed5003-80aa-4e19-aea3-5e24657030dc_isrForCrosstalkSources_996720_27.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/12ed5003-80aa-4e19-aea3-5e24657030dc_isrForCrosstalkSources_996720_27.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 678d44f2b516d9744e64a96e2ac9c5d1 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3e0c69c5-b8ba-4abc-84fb-6c5bdd7b9bbf_isrForCrosstalkSources_996709_26.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3e0c69c5-b8ba-4abc-84fb-6c5bdd7b9bbf_isrForCrosstalkSources_996709_26.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 80a7e4e6d1ab1032b3c2008254b3bea0 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/7285a442-faf5-46d1-9bea-49a8b4abfb0c_isrForCrosstalkSources_996720_53.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/7285a442-faf5-46d1-9bea-49a8b4abfb0c_isrForCrosstalkSources_996720_53.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash ebd8599a71cbaa6c2542e1268642238b parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/d5438126-c046-4533-a272-499256fe27e7_isrForCrosstalkSources_996715_36.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/d5438126-c046-4533-a272-499256fe27e7_isrForCrosstalkSources_996715_36.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash c3a95f7868137a7b8a91e9f378b46009 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/2970e918-3f5f-4c7a-a6e5-c9480e0085dc_isrForCrosstalkSources_996708_9.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/2970e918-3f5f-4c7a-a6e5-c9480e0085dc_isrForCrosstalkSources_996708_9.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash a0f4b442f3b77366d8a22e6ee363dca4 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2e48ec95-dbed-4fe5-88aa-b1e1f3ce2286_isrForCrosstalkSources_996712_6.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2e48ec95-dbed-4fe5-88aa-b1e1f3ce2286_isrForCrosstalkSources_996712_6.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash b12f2d9cf484460a165218cda3f06c84 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b56d6829-ab0d-492a-9fc7-39995c99383e_isrForCrosstalkSources_996716_21.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b56d6829-ab0d-492a-9fc7-39995c99383e_isrForCrosstalkSources_996716_21.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 661f98dddb688fa72b9c84da60ab6d73 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c2641b8d-5897-4799-ab40-3abb06494cc4_isrForCrosstalkSources_996708_48.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c2641b8d-5897-4799-ab40-3abb06494cc4_isrForCrosstalkSources_996708_48.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 3bb099ff9120689f51acc17ac5083bc0 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5bf7418d-24d5-4e67-b14c-c54f68949e73_isrForCrosstalkSources_996720_44.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5bf7418d-24d5-4e67-b14c-c54f68949e73_isrForCrosstalkSources_996720_44.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 44c4af1618b461ca2b6f387c784fe918 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/f6ddc6a2-15d9-40e1-b971-5af9d724827d_isrForCrosstalkSources_996718_16.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/f6ddc6a2-15d9-40e1-b971-5af9d724827d_isrForCrosstalkSources_996718_16.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 36806be52cb8729de8f1b05c6f156165 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c7ca80fb-40b5-4e47-ac9e-f6ed32a23f5b_isrForCrosstalkSources_996706_27.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c7ca80fb-40b5-4e47-ac9e-f6ed32a23f5b_isrForCrosstalkSources_996706_27.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash bf84963f2892e51039a0db262601e754 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6723bb06-aca1-4f52-8cd5-b3c338f0ff1e_isrForCrosstalkSources_996710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6723bb06-aca1-4f52-8cd5-b3c338f0ff1e_isrForCrosstalkSources_996710_34.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash a1665c51cca6fc04398d184915c756aa parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4570afea-4f9b-4123-a20a-d0a44f7580a0_isrForCrosstalkSources_996707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4570afea-4f9b-4123-a20a-d0a44f7580a0_isrForCrosstalkSources_996707_54.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 5ee5baea854da183dee10a361f243fab parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/d0669d97-c4a9-45b3-a615-0142202b03fe_isrForCrosstalkSources_996706_38.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/d0669d97-c4a9-45b3-a615-0142202b03fe_isrForCrosstalkSources_996706_38.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 205d9d59171007b002ea56ec3a692e08 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/f9a03880-bcb7-4641-816c-c7d1ac040f85_isrForCrosstalkSources_996718_47.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/f9a03880-bcb7-4641-816c-c7d1ac040f85_isrForCrosstalkSources_996718_47.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash fc66de61464c705a2026fa2851721c85 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/2cfd1818-a5c3-4fd6-91ca-34c38ef5e312_isrForCrosstalkSources_996706_54.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/2cfd1818-a5c3-4fd6-91ca-34c38ef5e312_isrForCrosstalkSources_996706_54.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash d451cd1246c2085a0c7c6e7f41d0d78c parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8906efb0-a19a-4c8d-bfd5-40c691802d43_isrForCrosstalkSources_996711_21.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8906efb0-a19a-4c8d-bfd5-40c691802d43_isrForCrosstalkSources_996711_21.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash a38f5091df75aa903295e2fa3af5d0f2 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/edb8cca6-4e2a-4f6a-bc76-5b0c93a30e44_isrForCrosstalkSources_996712_16.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/edb8cca6-4e2a-4f6a-bc76-5b0c93a30e44_isrForCrosstalkSources_996712_16.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash b5ba1509d661ee282cb9bad15bbc45b1 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/517e5c25-e45c-4154-8401-3885a20be71d_isrForCrosstalkSources_996719_50.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/517e5c25-e45c-4154-8401-3885a20be71d_isrForCrosstalkSources_996719_50.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 84f47be8f59587955120a32458fb23d3 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/23fa705c-bb80-4fbb-b0db-a7076c6d0bf4_isrForCrosstalkSources_996720_58.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/23fa705c-bb80-4fbb-b0db-a7076c6d0bf4_isrForCrosstalkSources_996720_58.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 152e2ccaf3706872669f5ae298c891a6 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e697dd83-0b04-4a48-afa1-78643e22d75d_isrForCrosstalkSources_996708_36.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e697dd83-0b04-4a48-afa1-78643e22d75d_isrForCrosstalkSources_996708_36.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash fc5883c96a2e3192fc9170bafa8383af parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c800ab2d-8207-4523-aa7d-b33d2dacc88e_isrForCrosstalkSources_996712_3.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c800ab2d-8207-4523-aa7d-b33d2dacc88e_isrForCrosstalkSources_996712_3.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash d43585b4c149c27fc7dc1ae531bd5fca parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/e53c10f1-0698-48a8-81e1-6b0eba9ff462_isrForCrosstalkSources_996712_49.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/e53c10f1-0698-48a8-81e1-6b0eba9ff462_isrForCrosstalkSources_996712_49.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash eb0af4f51d6decc708a2c82a0655dd9c parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/42bd9fd2-ab15-45be-bdb9-ce2c9858f3d9_isrForCrosstalkSources_996707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/42bd9fd2-ab15-45be-bdb9-ce2c9858f3d9_isrForCrosstalkSources_996707_20.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash eb8428046a0c8cfca9cbf74ebceb9bed parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f66134a8-b47f-49d1-bad7-440b5e7d5b1a_isrForCrosstalkSources_996712_56.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f66134a8-b47f-49d1-bad7-440b5e7d5b1a_isrForCrosstalkSources_996712_56.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 47a994a163449ba31dae19e87ed47ab4 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/cf148278-d0c6-4349-843d-147dca00b76a_isrForCrosstalkSources_996711_49.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/cf148278-d0c6-4349-843d-147dca00b76a_isrForCrosstalkSources_996711_49.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 01ebaf4cc43295423aeb97da97ad4db9 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/6eeefdad-17f3-4ff6-be4b-381a1d1bea1c_isrForCrosstalkSources_996705_19.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/6eeefdad-17f3-4ff6-be4b-381a1d1bea1c_isrForCrosstalkSources_996705_19.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash cb689fa1b75a9f349e475315fc77142d parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/678a033b-1396-492b-8115-28d43b7db879_isrForCrosstalkSources_996718_41.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/678a033b-1396-492b-8115-28d43b7db879_isrForCrosstalkSources_996718_41.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash cf6a8fc0da6ad2e598d1f8d2fa74f3f3 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cccf22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/1a846c31-affe-4909-ba41-5c3e8887a999_isrForCrosstalkSources_996710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/1a846c31-affe-4909-ba41-5c3e8887a999_isrForCrosstalkSources_996710_50.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash fe82ae5272f106fecb2a9c4b635b7699 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/1c353220-69a8-4525-9d84-5f1572cbf0bd_isrForCrosstalkSources_996713_19.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/1c353220-69a8-4525-9d84-5f1572cbf0bd_isrForCrosstalkSources_996713_19.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 84c6500d58b8f3ef78496efbbcb2e9e0 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b0417ab8-adde-4320-9f95-e51521f5f0c4_isrForCrosstalkSources_996705_49.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b0417ab8-adde-4320-9f95-e51521f5f0c4_isrForCrosstalkSources_996705_49.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 1404d8d0797477560349ed2e44bee5f1 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/8d2dfd76-7e8c-49bc-86a9-ca38573533ee_isrForCrosstalkSources_996714_5.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/8d2dfd76-7e8c-49bc-86a9-ca38573533ee_isrForCrosstalkSources_996714_5.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash f8c1512e7f77f96ab0b3a8d8a072f390 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e98873ca-bfb4-4f53-b5ce-2188a2890f23_isrForCrosstalkSources_996708_40.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e98873ca-bfb4-4f53-b5ce-2188a2890f23_isrForCrosstalkSources_996708_40.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 5edd8eaee0f55de6297ee3c0e9510375 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b2f66708-2b66-4886-aa31-470ccf10ed9e_isrForCrosstalkSources_996712_25.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b2f66708-2b66-4886-aa31-470ccf10ed9e_isrForCrosstalkSources_996712_25.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 0919f29433f0781fb4fe7b65838c46cc parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ad9632a2-5a65-4607-bd7d-237e5cb9b198_isrForCrosstalkSources_996720_47.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ad9632a2-5a65-4607-bd7d-237e5cb9b198_isrForCrosstalkSources_996720_47.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 52bbe484e84d714a5e5fcc75087093f6 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/9f0bd5a6-c36a-4709-9665-2f62348fb18e_isrForCrosstalkSources_996713_43.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/9f0bd5a6-c36a-4709-9665-2f62348fb18e_isrForCrosstalkSources_996713_43.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 1e29c753e37adc30281974c0da9dbd7c parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/fdb3c65d-22b6-47dd-aaa6-82fb26e092d7_isrForCrosstalkSources_996714_26.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/fdb3c65d-22b6-47dd-aaa6-82fb26e092d7_isrForCrosstalkSources_996714_26.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash c9aca2c62f07fa97990451c248a3b79e parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/36bf3096-8153-47c0-8c8e-4a6b47a02bf0_isrForCrosstalkSources_996711_28.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/36bf3096-8153-47c0-8c8e-4a6b47a02bf0_isrForCrosstalkSources_996711_28.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash db533fdbfdd892febb4bfd7f292131fc parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:46 proc_lsst.multi:170 [INFO] [multi] job_id 17071124 proc_lsst.multi INFO: [multi] job_id 17071124 2024-03-15 12:16:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071124 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071124 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17071124 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-15 12:16:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-15 12:16:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e7341843-bcd7-4e9b-8f28-10e9dd301cef_isrForCrosstalkSources_996707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e7341843-bcd7-4e9b-8f28-10e9dd301cef_isrForCrosstalkSources_996707_25.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 9d7b77b77b20ec0ce3aa19334d3506c8 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/1fec5e67-feeb-4b09-bc7c-f49f50bda9b1_isrForCrosstalkSources_996720_6.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/1fec5e67-feeb-4b09-bc7c-f49f50bda9b1_isrForCrosstalkSources_996720_6.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash ed0e7fb943d3c810b68a00c5c6be28ff parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/5c7d167e-b384-41d3-bb62-312032f5c353_isrForCrosstalkSources_996714_19.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/5c7d167e-b384-41d3-bb62-312032f5c353_isrForCrosstalkSources_996714_19.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash f45e7b7a73c3072f44e8374a8434d4f5 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/550f7ad4-ca6a-4564-ad03-c67b520246cc_isrForCrosstalkSources_996720_16.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/550f7ad4-ca6a-4564-ad03-c67b520246cc_isrForCrosstalkSources_996720_16.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash abf290f40e63ba4a54834151033d9594 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ddbc1b14-e28b-4c87-80d8-6e443d517d96_isrForCrosstalkSources_996707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ddbc1b14-e28b-4c87-80d8-6e443d517d96_isrForCrosstalkSources_996707_17.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash fa707a2f6c94bc37f7bebb4ae78db4ce parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c659d2c5-068d-4757-a529-69eaa6252721_isrForCrosstalkSources_996714_2.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c659d2c5-068d-4757-a529-69eaa6252721_isrForCrosstalkSources_996714_2.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 42c51915755e59a090260c78bb182570 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c944ae69-7dc1-4afd-9f89-1f51029b59f0_isrForCrosstalkSources_996714_27.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c944ae69-7dc1-4afd-9f89-1f51029b59f0_isrForCrosstalkSources_996714_27.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash d1b7ed9b1ddc9b2239f093f53df5310c parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8c5b1e4e-dfaa-4d41-8d15-049b08503fe2_isrForCrosstalkSources_996715_30.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8c5b1e4e-dfaa-4d41-8d15-049b08503fe2_isrForCrosstalkSources_996715_30.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 0af4332ed1ad11497caab29048da146e parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a75325d3-f207-4d5e-94c3-63bc6a1e6ef4_isrForCrosstalkSources_996712_27.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a75325d3-f207-4d5e-94c3-63bc6a1e6ef4_isrForCrosstalkSources_996712_27.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash fab55d584f6b21c77a69f33f7d1c6d0b parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f9a65cce-a4a7-48ca-be2a-f5e23ca7537f_isrForCrosstalkSources_996705_11.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f9a65cce-a4a7-48ca-be2a-f5e23ca7537f_isrForCrosstalkSources_996705_11.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash f1c3862ca9e77f40eef21d9059fb03f4 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/dc5d68d8-afa7-41f3-b12a-eb4119073240_isrForCrosstalkSources_996718_3.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/dc5d68d8-afa7-41f3-b12a-eb4119073240_isrForCrosstalkSources_996718_3.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 210119dd5da1fa851261bcb3a4c78a72 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ebff6dbd-c31a-4b7e-97e3-4b7dc9e4a5e6_isrForCrosstalkSources_996718_37.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ebff6dbd-c31a-4b7e-97e3-4b7dc9e4a5e6_isrForCrosstalkSources_996718_37.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash a0c35251435c12aa49dc503e2d385f58 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/18908587-af0e-4287-8f4d-648c2dd7fdd4_isrForCrosstalkSources_996709_19.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/18908587-af0e-4287-8f4d-648c2dd7fdd4_isrForCrosstalkSources_996709_19.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash d99994a761fd832811a01164b038914f parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2390a6a2-407a-4b6d-988c-c70b16a16ea2_isrForCrosstalkSources_996713_61.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2390a6a2-407a-4b6d-988c-c70b16a16ea2_isrForCrosstalkSources_996713_61.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 2edb4d72c42cf35093d6e222a293b17d parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/6db471f4-6508-475f-92da-7861386d2eff_isrForCrosstalkSources_996706_43.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/6db471f4-6508-475f-92da-7861386d2eff_isrForCrosstalkSources_996706_43.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 2f6fc612d17a60e21fd644be1a6961d9 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8e220760-6494-4809-b9d5-9eddfd6cb9a8_isrForCrosstalkSources_996712_53.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8e220760-6494-4809-b9d5-9eddfd6cb9a8_isrForCrosstalkSources_996712_53.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 0478c8fe44409971cbfd84abcefa73d9 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/396254cd-00ae-44f3-8e88-73d3d9df9a70_isrForCrosstalkSources_996711_57.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/396254cd-00ae-44f3-8e88-73d3d9df9a70_isrForCrosstalkSources_996711_57.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 5257ace4a2fa61ece72b68dca4ae1a9c parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/03b1d77e-0252-452f-ae7d-746ef9be58a0_isrForCrosstalkSources_996707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/03b1d77e-0252-452f-ae7d-746ef9be58a0_isrForCrosstalkSources_996707_6.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash d10df93ead263d64783b573c658a55a1 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/28e686c8-5be1-4826-923f-919f79d000e3_isrForCrosstalkSources_996714_20.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/28e686c8-5be1-4826-923f-919f79d000e3_isrForCrosstalkSources_996714_20.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 762ad349a84d26c587894402cb09f63a parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/04543c7d-1dda-4eff-961a-b9875d54b224_isrForCrosstalkSources_996714_39.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/04543c7d-1dda-4eff-961a-b9875d54b224_isrForCrosstalkSources_996714_39.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 22e8c64db817a7b1fd415673d8fe5e36 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/aa16f71b-95d5-4505-a844-ff0ee5bb01db_isrForCrosstalkSources_996718_1.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/aa16f71b-95d5-4505-a844-ff0ee5bb01db_isrForCrosstalkSources_996718_1.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 3fb4acb062c7ccc2549eda52003835f4 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5069c245-57d4-4f9f-b993-4d3fd0f0e523_isrForCrosstalkSources_996709_28.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5069c245-57d4-4f9f-b993-4d3fd0f0e523_isrForCrosstalkSources_996709_28.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash e7c87ea5dd5b7158e36b704e4b98ad0c parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e960b2a8-8e42-4eaa-806d-fc91016d04a8_isrForCrosstalkSources_996708_15.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e960b2a8-8e42-4eaa-806d-fc91016d04a8_isrForCrosstalkSources_996708_15.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 4ad7ae59801bc46805427ed156f3783e parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e628d147-f95f-4c6e-a87c-99a0c7b02e88_isrForCrosstalkSources_996721_42.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e628d147-f95f-4c6e-a87c-99a0c7b02e88_isrForCrosstalkSources_996721_42.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 790c0f65cea819d07808d8fed3e267c4 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/578428ec-f5a8-4fc4-8116-fec6a9fa5bfd_isrForCrosstalkSources_996717_54.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/578428ec-f5a8-4fc4-8116-fec6a9fa5bfd_isrForCrosstalkSources_996717_54.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 8dada57d5312d1ec357a73326bb91eb9 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7fe197b3-3980-4c2e-9cb7-ec1f82b238a5_isrForCrosstalkSources_996721_18.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7fe197b3-3980-4c2e-9cb7-ec1f82b238a5_isrForCrosstalkSources_996721_18.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 79457d4894828fc7446aad47b71f5242 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3df4386d-7fdf-4eaa-b457-f0cc7fc9f4d8_isrForCrosstalkSources_996711_51.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3df4386d-7fdf-4eaa-b457-f0cc7fc9f4d8_isrForCrosstalkSources_996711_51.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 0ddc2a27dfef81d7d8b0518dc8e1b250 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/5fc87a6f-05a6-465b-be05-93cba2ff574b_isrForCrosstalkSources_996712_11.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/5fc87a6f-05a6-465b-be05-93cba2ff574b_isrForCrosstalkSources_996712_11.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash c30be4e46c104c6af08cb8d4f5536327 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/13e74549-61ce-43a6-bf92-0f553a8b1d08_isrForCrosstalkSources_996705_1.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/13e74549-61ce-43a6-bf92-0f553a8b1d08_isrForCrosstalkSources_996705_1.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 0d282498bf38e9334f1bfcfe0d0e6557 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/679fd183-8559-4967-aa4f-67123a3c5eec_isrForCrosstalkSources_996720_11.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/679fd183-8559-4967-aa4f-67123a3c5eec_isrForCrosstalkSources_996720_11.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 6dd9e145b0a781fed8be0081c17491fd parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/02111ca9-4405-4590-9736-1de9c40325bd_isrForCrosstalkSources_996720_22.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/02111ca9-4405-4590-9736-1de9c40325bd_isrForCrosstalkSources_996720_22.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 01dd7eb5521fd3c009181f624d6a9c4a parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/cb700fae-c862-4ad1-b86d-4aec3bdf2385_isrForCrosstalkSources_996721_24.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/cb700fae-c862-4ad1-b86d-4aec3bdf2385_isrForCrosstalkSources_996721_24.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 47b49820af5a3f873801c6c7227878e6 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/169d5d68-8956-43d1-98fd-2e3b47615bb3_isrForCrosstalkSources_996713_7.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/169d5d68-8956-43d1-98fd-2e3b47615bb3_isrForCrosstalkSources_996713_7.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash c0ae39ada2a369088e3f893ab55edd86 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c0be4e60-4337-4ced-bfa0-7794dfa972ab_isrForCrosstalkSources_996705_4.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c0be4e60-4337-4ced-bfa0-7794dfa972ab_isrForCrosstalkSources_996705_4.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 88b7d43881968a013c2c3b097201a094 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ef8b8dbc-4b84-40f3-841f-83abf53a4a72_isrForCrosstalkSources_996713_35.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ef8b8dbc-4b84-40f3-841f-83abf53a4a72_isrForCrosstalkSources_996713_35.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 04f58d77ee83a9b561a8c72d344b4611 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7421d754-ac32-4a0c-836a-8cb587cc01ca_isrForCrosstalkSources_996712_8.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7421d754-ac32-4a0c-836a-8cb587cc01ca_isrForCrosstalkSources_996712_8.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash be158c9d00eae31baf49325353cf9883 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f7e7f7ac-7d38-4b48-a953-1b364507afcb_isrForCrosstalkSources_996710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f7e7f7ac-7d38-4b48-a953-1b364507afcb_isrForCrosstalkSources_996710_51.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 5cd4ab6a773e937cfdb7b813bde108bd parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/daa825ae-824f-4149-a0e0-e5a3a487f7cf_isrForCrosstalkSources_996713_24.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/daa825ae-824f-4149-a0e0-e5a3a487f7cf_isrForCrosstalkSources_996713_24.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 6506772ba629890c2a35c8adb3d9360a parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/56c91f8f-ebaa-49bc-8c69-e00be677245c_isrForCrosstalkSources_996708_46.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/56c91f8f-ebaa-49bc-8c69-e00be677245c_isrForCrosstalkSources_996708_46.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 42f42bbc80905eb7cbd8e75fd54746eb parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f4028fe0-4683-4e10-92e7-ef45c9ca0195_isrForCrosstalkSources_996705_17.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f4028fe0-4683-4e10-92e7-ef45c9ca0195_isrForCrosstalkSources_996705_17.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash e67e440ed0284d172ca795c185b28d00 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/008649da-7d7c-4e8d-b774-00dac9ae8de9_isrForCrosstalkSources_996713_44.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/008649da-7d7c-4e8d-b774-00dac9ae8de9_isrForCrosstalkSources_996713_44.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 9ac3c2b8b14af7b690bd012be3a859ce parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e048fe99-c717-4875-a9a2-1f84099b7677_isrForCrosstalkSources_996721_21.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e048fe99-c717-4875-a9a2-1f84099b7677_isrForCrosstalkSources_996721_21.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash e681053793d9996e97c61a227137a794 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e25ea7fa-1678-42a9-a5f2-798bf146c4d8_isrForCrosstalkSources_996720_41.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e25ea7fa-1678-42a9-a5f2-798bf146c4d8_isrForCrosstalkSources_996720_41.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 941487934e0014c67e5fb9f7484eb1af parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/87fba166-8af9-4827-a97b-e588a3df78fd_isrForCrosstalkSources_996712_21.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/87fba166-8af9-4827-a97b-e588a3df78fd_isrForCrosstalkSources_996712_21.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 99759b40e9a56b15e81678dc7b83e7bc parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/94bc3a17-9523-43ba-9a90-38204e069c68_isrForCrosstalkSources_996717_50.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/94bc3a17-9523-43ba-9a90-38204e069c68_isrForCrosstalkSources_996717_50.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 9d49711054369cf273b06dfe5f5d9af7 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/260c1a47-1a4a-4ade-ba17-094f3c376b65_isrForCrosstalkSources_996713_25.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/260c1a47-1a4a-4ade-ba17-094f3c376b65_isrForCrosstalkSources_996713_25.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 8ff1f79f6a43bfb7dd1baead5dc482a6 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/bc0d21f2-d32e-4f33-9a6c-99f7d6f5e89b_isrForCrosstalkSources_996717_11.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/bc0d21f2-d32e-4f33-9a6c-99f7d6f5e89b_isrForCrosstalkSources_996717_11.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 4cc26bc51ece49488c575364852af629 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/81ce262d-8152-4fc1-96b3-f5e97404d43c_isrForCrosstalkSources_996719_22.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/81ce262d-8152-4fc1-96b3-f5e97404d43c_isrForCrosstalkSources_996719_22.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 4cfdc902d63fe0f9d32866b88d3cd83e parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/dbf53692-7f21-4b33-ad44-01339cb8489a_isrForCrosstalkSources_996714_49.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/dbf53692-7f21-4b33-ad44-01339cb8489a_isrForCrosstalkSources_996714_49.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 624ad1e443127cd03d979b696c940175 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3bb25ed6-dd03-438b-80c7-e96bb8e62605_isrForCrosstalkSources_996720_4.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3bb25ed6-dd03-438b-80c7-e96bb8e62605_isrForCrosstalkSources_996720_4.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 6af601bedeaf748e4210243cbc9d2441 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6a94448a-affb-4d35-ad77-f329522388ac_isrForCrosstalkSources_996709_29.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6a94448a-affb-4d35-ad77-f329522388ac_isrForCrosstalkSources_996709_29.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash d10574b149cddd6a401c7915dae6a86a parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/5a6b7e98-6f9e-493b-8d76-16a94d481082_isrForCrosstalkSources_996715_39.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/5a6b7e98-6f9e-493b-8d76-16a94d481082_isrForCrosstalkSources_996715_39.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 1f7f5fb76af68c6101f75a85ab3a0439 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cc3f7a54-d1ae-4819-aec8-4a8781776178_isrForCrosstalkSources_996710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cc3f7a54-d1ae-4819-aec8-4a8781776178_isrForCrosstalkSources_996710_44.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash db66f03fff4ef06a91fbaecf7aa506e8 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/550e1b07-2234-4a74-9699-4c1a8b0a7ebf_isrForCrosstalkSources_996707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/550e1b07-2234-4a74-9699-4c1a8b0a7ebf_isrForCrosstalkSources_996707_35.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 892970fd10e3a1f63fc8a7e03a23d626 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/503bd415-06d3-4cec-9b8e-bb68c2a05b04_isrForCrosstalkSources_996711_8.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/503bd415-06d3-4cec-9b8e-bb68c2a05b04_isrForCrosstalkSources_996711_8.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 45912c9c32e1cbb98e42e9dee459d15d parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8e0654c9-3466-4bc2-9ca9-415aff27d445_isrForCrosstalkSources_996717_31.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8e0654c9-3466-4bc2-9ca9-415aff27d445_isrForCrosstalkSources_996717_31.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash a4195efe62eb33084365cb1d961420f2 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d8d498ba-d70a-47a8-be33-5a90a024ad02_isrForCrosstalkSources_996711_55.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d8d498ba-d70a-47a8-be33-5a90a024ad02_isrForCrosstalkSources_996711_55.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 761577e7887d68f5416422ee7095d7ed parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/76543138-7d27-48a7-a22d-36a6887ad526_isrForCrosstalkSources_996707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/76543138-7d27-48a7-a22d-36a6887ad526_isrForCrosstalkSources_996707_46.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash ba8ea5f06ce86f53b4065896cde4cffb parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/4bf875b0-9902-4571-a411-7a7975514ee1_isrForCrosstalkSources_996721_54.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/4bf875b0-9902-4571-a411-7a7975514ee1_isrForCrosstalkSources_996721_54.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 271743c167fbd2c0f7fa72f89782181b parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/d6997dcd-6da5-419f-b34c-c88d3e9f9d4f_isrForCrosstalkSources_996717_26.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/d6997dcd-6da5-419f-b34c-c88d3e9f9d4f_isrForCrosstalkSources_996717_26.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 5f2f8884bace4e75607dbad024169dc7 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a0f4c43e-e37f-4993-b03c-56bae108fdbf_isrForCrosstalkSources_996720_18.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a0f4c43e-e37f-4993-b03c-56bae108fdbf_isrForCrosstalkSources_996720_18.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 9e60def3befe8b74a632af3ce465c64a parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5e239d20-3bec-431c-a7ba-2bb9890ed98a_isrForCrosstalkSources_996707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5e239d20-3bec-431c-a7ba-2bb9890ed98a_isrForCrosstalkSources_996707_18.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 344dd211469d3ebd69bfa420781bec01 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:47 proc_lsst.multi:170 [INFO] [multi] job_id 17071125 proc_lsst.multi INFO: [multi] job_id 17071125 2024-03-15 12:16:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 12:16:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071125 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071125 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17071125 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-15 12:16:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-15 12:16:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 12:16:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/65dac767-3034-43c6-9129-374a3f169c3a_isrForCrosstalkSources_996715_26.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/65dac767-3034-43c6-9129-374a3f169c3a_isrForCrosstalkSources_996715_26.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash e543570d117febc3ed2c0efc06dd62bf parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9564b2b3-9fb4-4565-96eb-64720e4d373a_isrForCrosstalkSources_996718_53.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9564b2b3-9fb4-4565-96eb-64720e4d373a_isrForCrosstalkSources_996718_53.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 8c829690290e1f4610f2516915ebc58a parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/3cc834cc-bc21-4796-a710-f7a927231dc9_isrForCrosstalkSources_996715_46.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/3cc834cc-bc21-4796-a710-f7a927231dc9_isrForCrosstalkSources_996715_46.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash d64f514294b1e92f48172cb489af2fc1 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ceab6fb6-eb19-4833-a421-25889f8678bc_isrForCrosstalkSources_996705_52.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ceab6fb6-eb19-4833-a421-25889f8678bc_isrForCrosstalkSources_996705_52.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash dce3cd8a82a690c5d9784cdf1571bbc6 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6ff784ea-fd2a-4af5-9d22-2b296a616ac8_isrForCrosstalkSources_996711_22.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6ff784ea-fd2a-4af5-9d22-2b296a616ac8_isrForCrosstalkSources_996711_22.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 2ac0389e2a7c299fd7184b2fa377b16d parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8cc06a4e-6171-499f-85c8-5c8cc9e1ed72_isrForCrosstalkSources_996721_48.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8cc06a4e-6171-499f-85c8-5c8cc9e1ed72_isrForCrosstalkSources_996721_48.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash c6f73022399eb8b2ce9277d8488ad8ee parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/883b79ed-e1b9-4f21-b0ad-f80f1a3cf165_isrForCrosstalkSources_996709_15.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/883b79ed-e1b9-4f21-b0ad-f80f1a3cf165_isrForCrosstalkSources_996709_15.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 068cfd947c24bf0965e0f74ecc674a5f parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9d8c7646-955c-4fe7-b443-98f0abe85b2f_isrForCrosstalkSources_996708_53.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9d8c7646-955c-4fe7-b443-98f0abe85b2f_isrForCrosstalkSources_996708_53.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 7ee6835c37eeda207a84276baef35c5a parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fcedf03a-febc-47ba-b349-bfcee5180c84_isrForCrosstalkSources_996721_10.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fcedf03a-febc-47ba-b349-bfcee5180c84_isrForCrosstalkSources_996721_10.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 44ace196f488953c45befea0fa76d0c5 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/db0b1a01-004e-43ad-9b87-c2e91c8fe4fe_isrForCrosstalkSources_996721_62.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/db0b1a01-004e-43ad-9b87-c2e91c8fe4fe_isrForCrosstalkSources_996721_62.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 6adc84f502f4e73733e3f4afc094e828 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/bec4cedc-3ea4-48c5-a303-565f760f2448_isrForCrosstalkSources_996715_34.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/bec4cedc-3ea4-48c5-a303-565f760f2448_isrForCrosstalkSources_996715_34.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash d33f03668154fd8219d2f6b74119310c parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2340849b-eb90-445a-b3f4-7e84d73e17c7_isrForCrosstalkSources_996713_37.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2340849b-eb90-445a-b3f4-7e84d73e17c7_isrForCrosstalkSources_996713_37.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 45414a8dcd9af5ca0e35daeed839c7a5 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1dfafc43-95eb-43bd-8d45-c6bd67d5ccb1_isrForCrosstalkSources_996707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1dfafc43-95eb-43bd-8d45-c6bd67d5ccb1_isrForCrosstalkSources_996707_61.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash f9607780eae031a48d846c3064098087 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/96e96f1c-cc61-4abc-a7d0-c41b092d749d_isrForCrosstalkSources_996706_37.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/96e96f1c-cc61-4abc-a7d0-c41b092d749d_isrForCrosstalkSources_996706_37.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 62c806e7565cf58331e0cd78a3d3203b parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/46ce4d69-3548-47d3-9259-90c064e813fa_isrForCrosstalkSources_996708_34.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/46ce4d69-3548-47d3-9259-90c064e813fa_isrForCrosstalkSources_996708_34.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 8c494d2301888db250785477bda74543 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ce20cc28-5b54-4ef9-81e7-0b3e32b5dffc_isrForCrosstalkSources_996717_40.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ce20cc28-5b54-4ef9-81e7-0b3e32b5dffc_isrForCrosstalkSources_996717_40.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 5d611b32cf0d3f001c023f43353f96dd parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/1f3a0057-7a3d-4c0f-ac4e-58f6dcc41a25_isrForCrosstalkSources_996711_39.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/1f3a0057-7a3d-4c0f-ac4e-58f6dcc41a25_isrForCrosstalkSources_996711_39.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 9cbbdd1260e67dfc48cc06bf43429cbd parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/538c608c-2637-410d-9789-071b12d78a2b_isrForCrosstalkSources_996705_44.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/538c608c-2637-410d-9789-071b12d78a2b_isrForCrosstalkSources_996705_44.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash d4729893a0119ffc358c6d39f017c2e4 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/bb22b37d-6e3c-44d4-8604-341f2c595bde_isrForCrosstalkSources_996708_23.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/bb22b37d-6e3c-44d4-8604-341f2c595bde_isrForCrosstalkSources_996708_23.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash f6cca541aee49ab14d49fa9b197d1325 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/769531ae-9e3b-496e-a9a0-d1ecb0db871e_isrForCrosstalkSources_996717_28.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/769531ae-9e3b-496e-a9a0-d1ecb0db871e_isrForCrosstalkSources_996717_28.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash c3115fc4031b47e67708ed167b53c399 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/48ca19f9-b86d-4a0c-895e-2778a84d6a42_isrForCrosstalkSources_996711_1.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/48ca19f9-b86d-4a0c-895e-2778a84d6a42_isrForCrosstalkSources_996711_1.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 3aef9e998bf8022fb3863b19d0151bd8 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9d086772-6474-463d-9872-f3500baca8a8_isrForCrosstalkSources_996714_17.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9d086772-6474-463d-9872-f3500baca8a8_isrForCrosstalkSources_996714_17.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash ba16e2ebe5ba8b5647b9ec705ee19804 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3237d763-9188-4fcc-86bd-aea631f76c4d_isrForCrosstalkSources_996709_42.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3237d763-9188-4fcc-86bd-aea631f76c4d_isrForCrosstalkSources_996709_42.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash db9fb0c3d33244a61f51bb75e9deb657 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/f2f42c21-1c2d-4113-a60f-23fbbc5721e5_isrForCrosstalkSources_996715_3.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/f2f42c21-1c2d-4113-a60f-23fbbc5721e5_isrForCrosstalkSources_996715_3.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 8e52896689e3d0275c231a64072c7857 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7b22c0c1-5756-44cd-a60c-ad17bb2b2c34_isrForCrosstalkSources_996714_18.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7b22c0c1-5756-44cd-a60c-ad17bb2b2c34_isrForCrosstalkSources_996714_18.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash c7b37bff5cdfdba30d593d13cb7184da parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/cd4dcac9-afcf-423b-964c-11466eade8b3_isrForCrosstalkSources_996719_17.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/cd4dcac9-afcf-423b-964c-11466eade8b3_isrForCrosstalkSources_996719_17.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 8896548740f833af13704fabd59360a3 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c4b52bff-afa7-462c-9487-92f83d64912e_isrForCrosstalkSources_996714_38.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c4b52bff-afa7-462c-9487-92f83d64912e_isrForCrosstalkSources_996714_38.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 0112406bbc7ab9995c70e24e4bc825d0 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/17e5649e-4d3e-4158-b492-e573e9fe37f9_isrForCrosstalkSources_996719_3.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/17e5649e-4d3e-4158-b492-e573e9fe37f9_isrForCrosstalkSources_996719_3.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash c9e3b636abd32c97dc85b2ace7aeca5d parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b76915d2-fb71-46d9-9e82-6fad76d0dbd0_isrForCrosstalkSources_996706_21.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b76915d2-fb71-46d9-9e82-6fad76d0dbd0_isrForCrosstalkSources_996706_21.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash cee391da48219c41d13785f32e17ff4d parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/5bd0c617-cc93-4036-9016-c417227e7900_isrForCrosstalkSources_996712_36.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/5bd0c617-cc93-4036-9016-c417227e7900_isrForCrosstalkSources_996712_36.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash dc97c3db5be9cc0c875570a22de0ac6f parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/67d38442-8c5d-4e5d-831c-427d449e8fc7_isrForCrosstalkSources_996708_37.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/67d38442-8c5d-4e5d-831c-427d449e8fc7_isrForCrosstalkSources_996708_37.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 4776ea43c0fc537e5aea504916325d64 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77380e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/407fd2b5-add3-476d-a012-857c6a904f51_isrForCrosstalkSources_996705_47.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/407fd2b5-add3-476d-a012-857c6a904f51_isrForCrosstalkSources_996705_47.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 1e8c9bcfb3d2fd043b23f69933a7ccf2 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0fbe8343-0486-4cc7-a9c6-ecd486455490_isrForCrosstalkSources_996715_11.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0fbe8343-0486-4cc7-a9c6-ecd486455490_isrForCrosstalkSources_996715_11.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 507699e3241954244b35a03d23dfc3ed parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77382c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/0818ade6-a9d8-47ab-8503-63ef498e8576_isrForCrosstalkSources_996718_45.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/0818ade6-a9d8-47ab-8503-63ef498e8576_isrForCrosstalkSources_996718_45.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 60c43ffc527ae961a803d522daf02524 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/40fbba30-db8a-40ad-bb25-d657c0c21d5b_isrForCrosstalkSources_996706_16.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/40fbba30-db8a-40ad-bb25-d657c0c21d5b_isrForCrosstalkSources_996706_16.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 96aef32ba9af69a144a0fa76f3c9d311 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e582e5dc-9552-4dd2-8d9f-b58836194c0d_isrForCrosstalkSources_996708_1.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e582e5dc-9552-4dd2-8d9f-b58836194c0d_isrForCrosstalkSources_996708_1.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 6d56fc52c1f78b90687dd0e3caff6ddf parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ae7291b3-0240-49fe-a7e4-a9788e9782bb_isrForCrosstalkSources_996720_14.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ae7291b3-0240-49fe-a7e4-a9788e9782bb_isrForCrosstalkSources_996720_14.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash ffe7f2ffe64c3a84538b3aadaf23db21 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77384a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e98e5c0a-b18f-4589-af3a-13f24a156655_isrForCrosstalkSources_996720_10.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e98e5c0a-b18f-4589-af3a-13f24a156655_isrForCrosstalkSources_996720_10.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 2e601e4204b80ecca975564bf6ee100a parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77385e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c7c55d1a-97d4-49d3-b97e-322860db8c16_isrForCrosstalkSources_996719_33.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c7c55d1a-97d4-49d3-b97e-322860db8c16_isrForCrosstalkSources_996719_33.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 02f7e3be0aa55f46e27b9d65432a54e4 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8e3d6682-a99b-4bc0-a595-1ed8b4a6ccbd_isrForCrosstalkSources_996720_32.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8e3d6682-a99b-4bc0-a595-1ed8b4a6ccbd_isrForCrosstalkSources_996720_32.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash d37ffb8f67992c99a8916d411ed39023 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/038b3012-de99-402c-befb-0d132a52b982_isrForCrosstalkSources_996710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/038b3012-de99-402c-befb-0d132a52b982_isrForCrosstalkSources_996710_7.stderr 2024-03-15 12:16:48 proc_lsst.multi:170 [INFO] [multi] job_id 17071126 proc_lsst.multi INFO: [multi] job_id 17071126 2024-03-15 12:16:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 12:16:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071126 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071126 parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17071126 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.executors.status_handling INFO: Allocated block ID 4 parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-15 12:16:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-15 12:16:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 12:16:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash b7d677c15e971a9f9e7efdd39743969b parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/cf635695-b7c9-4448-b7d1-ee65b478c86f_isrForCrosstalkSources_996707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/cf635695-b7c9-4448-b7d1-ee65b478c86f_isrForCrosstalkSources_996707_44.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 463bd84c7f4f5d9a227409f41bdac76a parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a6ebab2e-f909-4d24-bf74-869eded5319b_isrForCrosstalkSources_996719_62.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a6ebab2e-f909-4d24-bf74-869eded5319b_isrForCrosstalkSources_996719_62.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 42d5fc566b9cad9862113060b70b3ad6 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd723ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/dbac4578-beca-4f8f-86fa-7ea1f52ef638_isrForCrosstalkSources_996712_1.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/dbac4578-beca-4f8f-86fa-7ea1f52ef638_isrForCrosstalkSources_996712_1.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 1a3c7ae31e4431b205a0157ee0dfb6b4 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508cd7236a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/9f11ad20-386c-43dc-9f4d-2b1606d7bbe3_isrForCrosstalkSources_996713_51.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/9f11ad20-386c-43dc-9f4d-2b1606d7bbe3_isrForCrosstalkSources_996713_51.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash a85b9232330c43d793dd227cc4596ed0 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/8c89a177-5adf-43f2-8a82-c7523082db3c_isrForCrosstalkSources_996705_61.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/8c89a177-5adf-43f2-8a82-c7523082db3c_isrForCrosstalkSources_996705_61.stderr parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash bb723690a15e8b750b2763175d75844b parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77387c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c0060344-2870-48df-bcd4-159f42c9cbc7_isrForCrosstalkSources_996705_53.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c0060344-2870-48df-bcd4-159f42c9cbc7_isrForCrosstalkSources_996705_53.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash f06e0e4983f87486a4f753dfa8576465 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/bd21885c-c950-4def-b497-d81856b34b4e_isrForCrosstalkSources_996714_51.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/bd21885c-c950-4def-b497-d81856b34b4e_isrForCrosstalkSources_996714_51.stderr parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash c08588a6ff54fba93d5c648ef83759b4 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/4309d474-6b01-4774-b4ba-e87dca62a745_isrForCrosstalkSources_996720_19.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/4309d474-6b01-4774-b4ba-e87dca62a745_isrForCrosstalkSources_996720_19.stderr parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 3f1eacd5b132ea43900eb11d09b21687 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a375dbb7-7c92-4232-85a6-f0d5f134d143_isrForCrosstalkSources_996713_54.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a375dbb7-7c92-4232-85a6-f0d5f134d143_isrForCrosstalkSources_996713_54.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 90e55e3cd3af5c427c0ff3ac238b9d1c parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/63d4811d-a7ec-4615-8449-fa40d1062a3c_isrForCrosstalkSources_996710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/63d4811d-a7ec-4615-8449-fa40d1062a3c_isrForCrosstalkSources_996710_45.stderr parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash a2d3a39bc4ab9c26efa575e281846a9a parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5992e02f-a5cb-4ea8-b1e6-3d8f29f385a7_isrForCrosstalkSources_996708_29.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5992e02f-a5cb-4ea8-b1e6-3d8f29f385a7_isrForCrosstalkSources_996708_29.stderr parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 3b9a70d3f3c2926c663c1ade52b746d5 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/2a220096-2a82-43e3-9810-a1d44b3963dc_isrForCrosstalkSources_996719_31.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/2a220096-2a82-43e3-9810-a1d44b3963dc_isrForCrosstalkSources_996719_31.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 81b0c237e9d38c1fa9563b9870efb120 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/1301a794-3121-4097-93d5-c2850d9ef646_isrForCrosstalkSources_996714_4.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/1301a794-3121-4097-93d5-c2850d9ef646_isrForCrosstalkSources_996714_4.stderr parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 05e7e68db306990aca2a2e7f318fbe54 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/2dec188b-f4eb-49f8-8819-7e449cf88b3a_isrForCrosstalkSources_996717_14.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/2dec188b-f4eb-49f8-8819-7e449cf88b3a_isrForCrosstalkSources_996717_14.stderr parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 9d74485165d04238b55cc6af5be1d020 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/091e59fd-7151-4fe6-9fef-5bc86a968bd2_isrForCrosstalkSources_996712_52.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/091e59fd-7151-4fe6-9fef-5bc86a968bd2_isrForCrosstalkSources_996712_52.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash e2b101330ae430f4a24ae38c5a2fbb44 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cad12935-7e23-4918-8951-9119944c8215_isrForCrosstalkSources_996714_45.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cad12935-7e23-4918-8951-9119944c8215_isrForCrosstalkSources_996714_45.stderr parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 9dfca0d76a73aa80b27277fe988295fc parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77391c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f6da887e-5bf4-4eb7-87e4-ca7da68e0d13_isrForCrosstalkSources_996709_21.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f6da887e-5bf4-4eb7-87e4-ca7da68e0d13_isrForCrosstalkSources_996709_21.stderr parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash f602185c1f3e1dfb3d094193800a5c25 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/7b96c66a-8cd4-4766-9248-5de8648eafbb_isrForCrosstalkSources_996706_55.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/7b96c66a-8cd4-4766-9248-5de8648eafbb_isrForCrosstalkSources_996706_55.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 68775510d82d2256db746bfa8aa7ba66 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/9d552b8c-b44d-4aa8-b85a-ebc71acb5da5_isrForCrosstalkSources_996709_53.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/9d552b8c-b44d-4aa8-b85a-ebc71acb5da5_isrForCrosstalkSources_996709_53.stderr parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 71c007f10af319993e44b6ae984926fd parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7152f469-7d2b-4d47-9443-b7bd729800bc_isrForCrosstalkSources_996714_7.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7152f469-7d2b-4d47-9443-b7bd729800bc_isrForCrosstalkSources_996714_7.stderr parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 01318759ad168932ef44df1d8f53aa17 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77393a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7bb06742-df2d-417f-a21e-e8fb7fabd701_isrForCrosstalkSources_996721_8.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7bb06742-df2d-417f-a21e-e8fb7fabd701_isrForCrosstalkSources_996721_8.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 22b84b1d71784a38c72b9f8f816f5612 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77394e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/ba09161d-15e5-4e1d-b8c3-4637e2a4aa00_isrForCrosstalkSources_996710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/ba09161d-15e5-4e1d-b8c3-4637e2a4aa00_isrForCrosstalkSources_996710_27.stderr parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 2fa9ffe8b4d0c1bf4087f08ad26a06ee parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/653edfed-c521-4b06-9bcf-99d7ca9412d2_isrForCrosstalkSources_996716_17.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/653edfed-c521-4b06-9bcf-99d7ca9412d2_isrForCrosstalkSources_996716_17.stderr parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 4a139e26a54d850be1b328a7d56749b6 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5c317e2f-e97f-46b1-bbb7-6a30db1abd7d_isrForCrosstalkSources_996718_38.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5c317e2f-e97f-46b1-bbb7-6a30db1abd7d_isrForCrosstalkSources_996718_38.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 40a4f6128378909e2c72d22f30a0ae7a parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e2d79c2d-2812-4969-89d9-7e41c6b61f9f_isrForCrosstalkSources_996706_35.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e2d79c2d-2812-4969-89d9-7e41c6b61f9f_isrForCrosstalkSources_996706_35.stderr parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash f8a9575ab3c9dc86f80a6174a7fc9071 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/eb5c23ab-4076-4b1f-be39-0fe14f0a51ed_isrForCrosstalkSources_996713_5.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/eb5c23ab-4076-4b1f-be39-0fe14f0a51ed_isrForCrosstalkSources_996713_5.stderr parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 37e09542617424651206e9c0aabf0189 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4960afbb-a2ad-4f76-9c4d-d7fea7bd7f5e_isrForCrosstalkSources_996718_46.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4960afbb-a2ad-4f76-9c4d-d7fea7bd7f5e_isrForCrosstalkSources_996718_46.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash a4121341fb99bcbea23bce9a859e9466 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/98cacf16-200e-47e3-b5b4-d481744c5be5_isrForCrosstalkSources_996719_26.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/98cacf16-200e-47e3-b5b4-d481744c5be5_isrForCrosstalkSources_996719_26.stderr parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 126c444d7df84bd872336bd8b56b3c43 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77398a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/cffca9eb-01ab-4481-8f04-b6ffce2ed6e8_isrForCrosstalkSources_996706_31.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/cffca9eb-01ab-4481-8f04-b6ffce2ed6e8_isrForCrosstalkSources_996706_31.stderr parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 1e3c846834385ed9ed64df2555e277ee parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/267de4a7-61d3-47c3-9f62-13b68ae81503_isrForCrosstalkSources_996713_45.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/267de4a7-61d3-47c3-9f62-13b68ae81503_isrForCrosstalkSources_996713_45.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash bf799d0cb4f0a594da0cab5c2022655f parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/5916108f-f466-4efb-b766-8f06c9148b37_isrForCrosstalkSources_996716_58.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/5916108f-f466-4efb-b766-8f06c9148b37_isrForCrosstalkSources_996716_58.stderr parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 5de27ec9971fd91801c75b127cb57e99 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77399e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3ac79982-a432-4dd8-9e17-0b794e9b6457_isrForCrosstalkSources_996706_6.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3ac79982-a432-4dd8-9e17-0b794e9b6457_isrForCrosstalkSources_996706_6.stderr parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash e0f73aec8736c412ea02e896a6783ca0 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/fc0e34c2-1533-4d2e-8171-3fec59ac717a_isrForCrosstalkSources_996708_17.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/fc0e34c2-1533-4d2e-8171-3fec59ac717a_isrForCrosstalkSources_996708_17.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 21291f1d37a3cfb5171ca09a02e35d51 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/421523cc-fc66-4e7c-9b39-aa57a9ccbf38_isrForCrosstalkSources_996716_49.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/421523cc-fc66-4e7c-9b39-aa57a9ccbf38_isrForCrosstalkSources_996716_49.stderr parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 09bdc55daafe2c46513e9fd23cebb065 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/71eb4b27-a458-489c-a8bd-6eeba9ca962d_isrForCrosstalkSources_996717_39.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/71eb4b27-a458-489c-a8bd-6eeba9ca962d_isrForCrosstalkSources_996717_39.stderr parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash dfa1ed62d43762b41fedbf9ec677290b parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/c713270f-7045-4629-831f-a489bf2e71af_isrForCrosstalkSources_996717_16.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/c713270f-7045-4629-831f-a489bf2e71af_isrForCrosstalkSources_996717_16.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 2362a6a5227b8eb0420610f4efe0783a parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c69df5fb-d7fd-4ca7-bfc5-c1ef6f981dc2_isrForCrosstalkSources_996712_26.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c69df5fb-d7fd-4ca7-bfc5-c1ef6f981dc2_isrForCrosstalkSources_996712_26.stderr parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 343bf80cc55a8873603f5da90162e605 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e777e4f2-1827-4ffc-95bc-4f6a31dedb14_isrForCrosstalkSources_996715_15.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e777e4f2-1827-4ffc-95bc-4f6a31dedb14_isrForCrosstalkSources_996715_15.stderr parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 4cf5813890e8948b0114f22a2d017b52 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7c1235ed-3d1e-42bd-9a8c-33be8fcd6d29_isrForCrosstalkSources_996714_24.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7c1235ed-3d1e-42bd-9a8c-33be8fcd6d29_isrForCrosstalkSources_996714_24.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 74017942e45eb168dc480f15a026184f parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7739ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8948b12f-af9a-4a03-a387-7e3c3e726af6_isrForCrosstalkSources_996717_44.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8948b12f-af9a-4a03-a387-7e3c3e726af6_isrForCrosstalkSources_996717_44.stderr parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 6b231eb0d0d0cba93684d49e846fcbaf parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/48ca5fa6-6e3e-46c4-9b92-fa22ca47dd01_isrForCrosstalkSources_996717_42.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/48ca5fa6-6e3e-46c4-9b92-fa22ca47dd01_isrForCrosstalkSources_996717_42.stderr parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 5b55d19c35e00e2e7d3f1a5c6cef8464 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/8be162bb-3e48-497b-99d8-ced93a431739_isrForCrosstalkSources_996708_61.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/8be162bb-3e48-497b-99d8-ced93a431739_isrForCrosstalkSources_996708_61.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 9eb271928c4b1b174d80d5feb53bdef3 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:49 proc_lsst.multi:170 [INFO] [multi] job_id 17071127 proc_lsst.multi INFO: [multi] job_id 17071127 2024-03-15 12:16:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 12:16:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071127 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071127 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17071127 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-15 12:16:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-15 12:16:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 12:16:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1d6a1b2b-7376-4d0d-bb2e-cd49b7d5aac3_isrForCrosstalkSources_996719_11.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1d6a1b2b-7376-4d0d-bb2e-cd49b7d5aac3_isrForCrosstalkSources_996719_11.stderr parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash e8011101a101c8987a1db7573f9775c7 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/41c73bb0-705a-4102-ba7d-2bead7b61316_isrForCrosstalkSources_996716_9.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/41c73bb0-705a-4102-ba7d-2bead7b61316_isrForCrosstalkSources_996716_9.stderr parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 92d10a54ab3af1483ffe59122e66db08 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77389a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/618f55db-81f8-438f-9590-0d00d90e8ea8_isrForCrosstalkSources_996708_44.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/618f55db-81f8-438f-9590-0d00d90e8ea8_isrForCrosstalkSources_996708_44.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 0004b1aa5a0ec4e7423c66545dbef3e5 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/348f4eff-3e0a-4b8b-9a5f-64a3471c9ed7_isrForCrosstalkSources_996705_32.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/348f4eff-3e0a-4b8b-9a5f-64a3471c9ed7_isrForCrosstalkSources_996705_32.stderr parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash ad9eadbe032430bef52b64a933537a0e parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/50c449c0-f854-4d42-869b-1d42f3d4f91a_isrForCrosstalkSources_996708_22.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/50c449c0-f854-4d42-869b-1d42f3d4f91a_isrForCrosstalkSources_996708_22.stderr parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 1abc080be91211f1ea22f702999f601f parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/e081d6b0-c102-47b4-9bf3-45be8aeced66_isrForCrosstalkSources_996719_16.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/e081d6b0-c102-47b4-9bf3-45be8aeced66_isrForCrosstalkSources_996719_16.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash e2085db765fc11c1012750ddc350994b parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/76eb361d-fec9-43df-b266-2bce1fd1ac31_isrForCrosstalkSources_996721_23.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/76eb361d-fec9-43df-b266-2bce1fd1ac31_isrForCrosstalkSources_996721_23.stderr parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 701ff33c3adb4313a319a2b242ccb8a2 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/fe2a3f56-b9e4-46ea-ab62-ac0ab98663b8_isrForCrosstalkSources_996708_32.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/fe2a3f56-b9e4-46ea-ab62-ac0ab98663b8_isrForCrosstalkSources_996708_32.stderr parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash b010ad272d0491c2c3d2f61d6625ac39 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/88e66b53-2865-4d6a-85ed-3facd1e2c7dc_isrForCrosstalkSources_996721_27.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/88e66b53-2865-4d6a-85ed-3facd1e2c7dc_isrForCrosstalkSources_996721_27.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash ae6f8345baadd69c50099b34d123eb07 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3aca22c1-f95d-492b-9b3f-18885944c9ea_isrForCrosstalkSources_996711_44.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3aca22c1-f95d-492b-9b3f-18885944c9ea_isrForCrosstalkSources_996711_44.stderr parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 5a606fbbc82dfdf973abf3b487d5ecde parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f505f6d8-d7fe-46be-a28f-021e55c4b523_isrForCrosstalkSources_996716_23.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f505f6d8-d7fe-46be-a28f-021e55c4b523_isrForCrosstalkSources_996716_23.stderr parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash d9683d7d00ff8821c2e3fe1e89df78fb parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/80ce1adc-f4be-47cd-b12c-47837d84ccc8_isrForCrosstalkSources_996709_49.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/80ce1adc-f4be-47cd-b12c-47837d84ccc8_isrForCrosstalkSources_996709_49.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash fd8d46129de6e48243b7c4e545657e00 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e55eef4f-e75a-46f2-b1ee-ff27360b9b8a_isrForCrosstalkSources_996720_37.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e55eef4f-e75a-46f2-b1ee-ff27360b9b8a_isrForCrosstalkSources_996720_37.stderr parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 8e003bc22d95033612e5a930a2f9a196 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/57498ec4-983b-4fca-8976-562268c734f2_isrForCrosstalkSources_996718_9.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/57498ec4-983b-4fca-8976-562268c734f2_isrForCrosstalkSources_996718_9.stderr parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 549fb23cdcdcd48bc70f1901f2e58a5c parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8632dceb-5f03-49e2-9b34-fe7d3719f427_isrForCrosstalkSources_996721_53.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8632dceb-5f03-49e2-9b34-fe7d3719f427_isrForCrosstalkSources_996721_53.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 18afeb0cb7b8193bd76be192c51bf077 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9d390578-c69e-457b-8a48-2fede5059f5b_isrForCrosstalkSources_996708_18.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9d390578-c69e-457b-8a48-2fede5059f5b_isrForCrosstalkSources_996708_18.stderr parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash e62d19041f2ab81a08a976d17330f0f3 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/0561efe3-bae1-4dd0-8c8f-56f1a590c4dc_isrForCrosstalkSources_996714_50.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/0561efe3-bae1-4dd0-8c8f-56f1a590c4dc_isrForCrosstalkSources_996714_50.stderr parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash fcd9001da3a75e16f0fe8c3ff87e9215 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/12f204d1-e104-418e-bd9b-236667ba36ea_isrForCrosstalkSources_996719_10.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/12f204d1-e104-418e-bd9b-236667ba36ea_isrForCrosstalkSources_996719_10.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 4bd9d38b9900971189842554ba6e10a5 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/aa0159ca-5e8a-4137-8c2a-38375a49097f_isrForCrosstalkSources_996705_3.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/aa0159ca-5e8a-4137-8c2a-38375a49097f_isrForCrosstalkSources_996705_3.stderr parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 4c8d790f0ac8a4d06310fc7b8c3903f4 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/80638d52-9011-485e-b53b-c06c2ab97052_isrForCrosstalkSources_996716_18.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/80638d52-9011-485e-b53b-c06c2ab97052_isrForCrosstalkSources_996716_18.stderr parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 053ea9222e5455955cb3d0567422cdcc parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6af3515f-2b66-445a-9c22-98ef8503687c_isrForCrosstalkSources_996709_34.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6af3515f-2b66-445a-9c22-98ef8503687c_isrForCrosstalkSources_996709_34.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 1516622b485a1a06229d964c6813735f parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5a5f249c-c6fc-4c3c-ad21-a2bac68cf970_isrForCrosstalkSources_996719_15.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5a5f249c-c6fc-4c3c-ad21-a2bac68cf970_isrForCrosstalkSources_996719_15.stderr parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 5c8c46d5a31ef207dbd25f840eca3b1a parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fee7b79a-5844-4321-858f-fa7109531a21_isrForCrosstalkSources_996705_27.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fee7b79a-5844-4321-858f-fa7109531a21_isrForCrosstalkSources_996705_27.stderr parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash ec98e1a4944df5e035961701a09d8d64 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3e000eb2-b1f5-4e69-a3f7-0167ffc48013_isrForCrosstalkSources_996705_9.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3e000eb2-b1f5-4e69-a3f7-0167ffc48013_isrForCrosstalkSources_996705_9.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash cb919412763834527b6bd5ad9de3dfdf parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a3a13918-16ed-4f8a-8cdd-f6b5c917a6f5_isrForCrosstalkSources_996709_52.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a3a13918-16ed-4f8a-8cdd-f6b5c917a6f5_isrForCrosstalkSources_996709_52.stderr parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash cd2c709d99997b11037923faed726fdc parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/6913f9f4-b476-4e37-8c72-cbe61f71919d_isrForCrosstalkSources_996717_15.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/6913f9f4-b476-4e37-8c72-cbe61f71919d_isrForCrosstalkSources_996717_15.stderr parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 84945da44d2185d502fcc4f638136e69 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0d4f7a63-dc6b-43ce-a27a-06739b3594fb_isrForCrosstalkSources_996720_30.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0d4f7a63-dc6b-43ce-a27a-06739b3594fb_isrForCrosstalkSources_996720_30.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 8b9925144b2a1f4dc64bdb8b040ea0b7 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/96116b52-4360-46a0-b302-8fecaa20ae4a_isrForCrosstalkSources_996709_44.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/96116b52-4360-46a0-b302-8fecaa20ae4a_isrForCrosstalkSources_996709_44.stderr parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 63e01d7e0be8c56be45ca2d81f196479 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/d33a9b46-3b1f-4f34-8c74-10c0f048b19c_isrForCrosstalkSources_996714_62.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/d33a9b46-3b1f-4f34-8c74-10c0f048b19c_isrForCrosstalkSources_996714_62.stderr parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 6c7a9f87014345ae1449a1f53cce98bc parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/07cec972-c502-442b-a670-87d0a52ffa13_isrForCrosstalkSources_996716_34.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/07cec972-c502-442b-a670-87d0a52ffa13_isrForCrosstalkSources_996716_34.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 757baf3e3093bda7b64db5f53f94a5fd parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fe0f1f71-fec8-46e8-b26e-260c7420b3f0_isrForCrosstalkSources_996705_62.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fe0f1f71-fec8-46e8-b26e-260c7420b3f0_isrForCrosstalkSources_996705_62.stderr parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 7d9700bafc0dad65d27143d00a9574cc parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/bdbad49e-4eb0-4349-9fc1-d844245c78fc_isrForCrosstalkSources_996719_60.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/bdbad49e-4eb0-4349-9fc1-d844245c78fc_isrForCrosstalkSources_996719_60.stderr parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 2a3f620aa76f6185071d55568a8fe6a4 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2aeb3d99-bedf-4230-ad36-2e93052891ea_isrForCrosstalkSources_996714_25.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2aeb3d99-bedf-4230-ad36-2e93052891ea_isrForCrosstalkSources_996714_25.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash bf3ff9071a7a158648e0e08478370b16 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/14b2cbe8-ce03-4d1c-89a9-2dd990c72a92_isrForCrosstalkSources_996716_19.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/14b2cbe8-ce03-4d1c-89a9-2dd990c72a92_isrForCrosstalkSources_996716_19.stderr parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash dc0541be4a32b1200d60bdb5aad7e50a parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6ff6882f-e8ca-413a-9ef6-7c7cf8d4e98c_isrForCrosstalkSources_996710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6ff6882f-e8ca-413a-9ef6-7c7cf8d4e98c_isrForCrosstalkSources_996710_5.stderr parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash e210bca4abcc6ffc765241a8bb80dd3e parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ebe97a4c-a7de-486e-9f9c-9afec502f7d0_isrForCrosstalkSources_996718_4.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ebe97a4c-a7de-486e-9f9c-9afec502f7d0_isrForCrosstalkSources_996718_4.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 867889f818eda809645155ebfbce1e84 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/320823f3-1bb7-4777-943f-aa0258e8d7b8_isrForCrosstalkSources_996713_59.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/320823f3-1bb7-4777-943f-aa0258e8d7b8_isrForCrosstalkSources_996713_59.stderr parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash c3f3b6d25a0df6806926c1c7dc7af824 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/80c66cbd-47df-4cd9-9b62-ac5d88295495_isrForCrosstalkSources_996713_32.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/80c66cbd-47df-4cd9-9b62-ac5d88295495_isrForCrosstalkSources_996713_32.stderr parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash dff1af7f5b3047e67252d52fe9562ccf parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/51099594-147d-4d80-918a-4551afb4e76b_isrForCrosstalkSources_996707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/51099594-147d-4d80-918a-4551afb4e76b_isrForCrosstalkSources_996707_8.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 13e86d808f9aaf55eacfb5a82513ba24 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/a084ff6f-22fc-44cc-9654-712e15877cfc_isrForCrosstalkSources_996715_43.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/a084ff6f-22fc-44cc-9654-712e15877cfc_isrForCrosstalkSources_996715_43.stderr parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 63fb649516fb02d252f3735f9bbbe78f parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5b1a94a9-f7d1-4149-bee9-e7bf6f1dd127_isrForCrosstalkSources_996721_36.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5b1a94a9-f7d1-4149-bee9-e7bf6f1dd127_isrForCrosstalkSources_996721_36.stderr parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 6db452b9478425cb5ddd382fbcc0088c parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/25d9eabc-da48-486d-b0ca-0b436ac51816_isrForCrosstalkSources_996707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/25d9eabc-da48-486d-b0ca-0b436ac51816_isrForCrosstalkSources_996707_60.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 31b747048f03a93f3458c395d5987608 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e43ef72f-ffbb-47c8-93d9-c0268739c512_isrForCrosstalkSources_996708_31.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e43ef72f-ffbb-47c8-93d9-c0268739c512_isrForCrosstalkSources_996708_31.stderr parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 5eba88e060f6f2983ccb39ae7b02f602 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/60529b3c-29c3-4a0b-80d0-883fa78c7b1b_isrForCrosstalkSources_996717_57.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/60529b3c-29c3-4a0b-80d0-883fa78c7b1b_isrForCrosstalkSources_996717_57.stderr parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 09ad31e1b5031cf02a159d68484c57f4 parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/0c203e92-afc3-4f3a-be98-eee6f7cc19bc_isrForCrosstalkSources_996721_45.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/0c203e92-afc3-4f3a-be98-eee6f7cc19bc_isrForCrosstalkSources_996721_45.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash e4a8dc2ecbb86ade8826a873f704898a parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8f05ad04-f7d0-4f34-990e-c204aa2939cd_isrForCrosstalkSources_996718_10.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8f05ad04-f7d0-4f34-990e-c204aa2939cd_isrForCrosstalkSources_996718_10.stderr parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 1414670f38583e75f44d910d47ed38fd parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ffc9149f-90e0-4856-b332-ddf301fe109f_isrForCrosstalkSources_996706_49.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ffc9149f-90e0-4856-b332-ddf301fe109f_isrForCrosstalkSources_996706_49.stderr parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 1a09d14c30f1d3df2d182eee78c2898e parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7738040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2259e3c9-386b-471d-b47a-13fd79835aa1_isrForCrosstalkSources_996710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2259e3c9-386b-471d-b47a-13fd79835aa1_isrForCrosstalkSources_996710_41.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash a6d2254dc14237632040865ebdc1c8f5 parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f5fb4d68-f423-4aac-a505-efffda43a5f2_isrForCrosstalkSources_996711_3.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f5fb4d68-f423-4aac-a505-efffda43a5f2_isrForCrosstalkSources_996711_3.stderr parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash c0d8e22be2a3e61b713691be8c73cf47 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fc625098-375a-4d56-a377-4f5703d880a5_isrForCrosstalkSources_996705_28.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fc625098-375a-4d56-a377-4f5703d880a5_isrForCrosstalkSources_996705_28.stderr parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash db66909c25535a4f3f3e532d22ebd0eb parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9075fb3f-4c48-4fed-8e54-1eebf8424c0b_isrForCrosstalkSources_996706_15.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9075fb3f-4c48-4fed-8e54-1eebf8424c0b_isrForCrosstalkSources_996706_15.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 8664f6604a16d088806f9169378d4f67 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77900e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dc43bfd7-6654-48f3-884e-5fc6d8e67bca_isrForCrosstalkSources_996711_32.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dc43bfd7-6654-48f3-884e-5fc6d8e67bca_isrForCrosstalkSources_996711_32.stderr parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 67683ca6856bf5c2416f1ecdf2b9dc3d parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8766a960-c236-420e-a99d-db7acf344b07_isrForCrosstalkSources_996712_2.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8766a960-c236-420e-a99d-db7acf344b07_isrForCrosstalkSources_996712_2.stderr parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 6e9bdadcf31fef087e98f3fa3bd3713c parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77902c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8e1af386-5650-4d01-a067-d9a397ee036f_isrForCrosstalkSources_996712_20.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8e1af386-5650-4d01-a067-d9a397ee036f_isrForCrosstalkSources_996712_20.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash eb9ce7a0113362ec5f3af8145ecca3c1 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c1167a60-7767-4863-85ab-e3bfc9dfb853_isrForCrosstalkSources_996719_58.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c1167a60-7767-4863-85ab-e3bfc9dfb853_isrForCrosstalkSources_996719_58.stderr parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 7f768389d9e77c40784a04101d159f8a parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77904a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/674017d1-a186-4f34-b4d3-3c45345241e3_isrForCrosstalkSources_996716_41.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/674017d1-a186-4f34-b4d3-3c45345241e3_isrForCrosstalkSources_996716_41.stderr parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash a3ba8a7a95a50cdf817f273e27e1b300 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5da6bcb8-fd77-417e-9d5a-89053043879a_isrForCrosstalkSources_996718_55.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5da6bcb8-fd77-417e-9d5a-89053043879a_isrForCrosstalkSources_996718_55.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 4fb9ed95d70c6155d9a52e582592cb80 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1161420e-515a-4d78-a407-f3218aa43fe0_isrForCrosstalkSources_996719_23.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1161420e-515a-4d78-a407-f3218aa43fe0_isrForCrosstalkSources_996719_23.stderr parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 8ab36a0c6a75df148fe0ef9ad8e34e85 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77905e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/fdb2f32f-b8eb-41df-8b7d-317fb81e846f_isrForCrosstalkSources_996720_9.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/fdb2f32f-b8eb-41df-8b7d-317fb81e846f_isrForCrosstalkSources_996720_9.stderr parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 9397dc4528b4249daabca798a5137e4a parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/9768af4c-7c9c-4259-8d02-d1eab997d8b7_isrForCrosstalkSources_996715_1.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/9768af4c-7c9c-4259-8d02-d1eab997d8b7_isrForCrosstalkSources_996715_1.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash b99d6d3c6f8cae80a9c84965c8aa3443 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77907c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0d4eae95-cf15-4853-b57e-d22ca644410c_isrForCrosstalkSources_996715_42.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0d4eae95-cf15-4853-b57e-d22ca644410c_isrForCrosstalkSources_996715_42.stderr parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash c117c829aea4aea9491d0aed5827323e parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e7b8af98-10c1-4086-8d0b-907604259740_isrForCrosstalkSources_996715_49.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e7b8af98-10c1-4086-8d0b-907604259740_isrForCrosstalkSources_996715_49.stderr parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash c25688f3b298b3c9541b2309ff808d05 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/e831800d-f695-422c-9939-ccbdde843214_isrForCrosstalkSources_996714_21.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/e831800d-f695-422c-9939-ccbdde843214_isrForCrosstalkSources_996714_21.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 618db001481a0771fbed6bd0ca7ed814 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d317cfad-afc8-48ec-bec9-c567cbe78c19_isrForCrosstalkSources_996705_55.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d317cfad-afc8-48ec-bec9-c567cbe78c19_isrForCrosstalkSources_996705_55.stderr parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 040400a9b54866526581836e45527a24 parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77909a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4b7a88b1-702e-4b4b-8a8e-f79b5453d945_isrForCrosstalkSources_996719_20.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4b7a88b1-702e-4b4b-8a8e-f79b5453d945_isrForCrosstalkSources_996719_20.stderr parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash e39830cc05fef4e6623346c7ec3aeba2 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:49 proc_lsst.multi:170 [INFO] [multi] job_id 17071128 proc_lsst.multi INFO: [multi] job_id 17071128 2024-03-15 12:16:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:16:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071128 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071128 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17071128 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-15 12:16:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-15 12:16:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:16:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed4af404-c071-4797-ac42-c36634cd5ec0_isrForCrosstalkSources_996709_46.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed4af404-c071-4797-ac42-c36634cd5ec0_isrForCrosstalkSources_996709_46.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash dcae5e961db1ce17f9a02c3b6c06a924 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/7d38e4ea-e6fe-4bd3-ab38-a491ea8554fe_isrForCrosstalkSources_996710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/7d38e4ea-e6fe-4bd3-ab38-a491ea8554fe_isrForCrosstalkSources_996710_10.stderr parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 1bc1cddfbd795262de2ac6878e4d8154 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/85d73c33-ba28-46a4-9faf-3b7f3e819e74_isrForCrosstalkSources_996711_50.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/85d73c33-ba28-46a4-9faf-3b7f3e819e74_isrForCrosstalkSources_996711_50.stderr parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 1e1c3d17d7ca0333667e7c3a4db66f48 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/408d5709-6480-4a33-898c-d3ac59fe0210_isrForCrosstalkSources_996714_31.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/408d5709-6480-4a33-898c-d3ac59fe0210_isrForCrosstalkSources_996714_31.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash b6353a576426c086220753581643a7a0 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b01532a8-3f9d-455d-ade8-a12ed24798a1_isrForCrosstalkSources_996705_35.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b01532a8-3f9d-455d-ade8-a12ed24798a1_isrForCrosstalkSources_996705_35.stderr parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash d6a59605fb739fd27eebbba5ed5eae03 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/df7a351e-d4d9-45a9-829c-48aeb253063b_isrForCrosstalkSources_996720_21.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/df7a351e-d4d9-45a9-829c-48aeb253063b_isrForCrosstalkSources_996720_21.stderr parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 01e0c441ead883ff7a5ce75e9ace16e6 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/eb5dc226-e578-4975-9ac6-3c52860defd9_isrForCrosstalkSources_996705_45.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/eb5dc226-e578-4975-9ac6-3c52860defd9_isrForCrosstalkSources_996705_45.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash e8d4e68afd9139a130dea46b63f1c913 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3e89ebbf-7f22-49d4-921f-68529b4ef50c_isrForCrosstalkSources_996707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3e89ebbf-7f22-49d4-921f-68529b4ef50c_isrForCrosstalkSources_996707_33.stderr parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 106879f14d855567790400ce6bfbe4e8 parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9b904e8e-bafa-416b-8d10-ee0a4669403f_isrForCrosstalkSources_996718_50.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9b904e8e-bafa-416b-8d10-ee0a4669403f_isrForCrosstalkSources_996718_50.stderr parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 25579499852f85da383aff88413a3df3 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/de3085cb-5a1d-432f-aa0e-25bdcdafb20f_isrForCrosstalkSources_996715_61.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/de3085cb-5a1d-432f-aa0e-25bdcdafb20f_isrForCrosstalkSources_996715_61.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash be148b18f197f45633b0ab9eb6c2aba9 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/b62416e2-384b-4c1c-9fff-410e96d94ab2_isrForCrosstalkSources_996720_12.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/b62416e2-384b-4c1c-9fff-410e96d94ab2_isrForCrosstalkSources_996720_12.stderr parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 3d98cc3dc5953224dd30188dbb3db4d4 parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/9c388c2c-92ec-4e3c-8219-2c76d35549ba_isrForCrosstalkSources_996709_35.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/9c388c2c-92ec-4e3c-8219-2c76d35549ba_isrForCrosstalkSources_996709_35.stderr parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 370cdb2e1a6c994347a7901a9d1cb458 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/cc47aa71-b156-448a-84d8-76ff2d242f41_isrForCrosstalkSources_996721_37.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/cc47aa71-b156-448a-84d8-76ff2d242f41_isrForCrosstalkSources_996721_37.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash de0bb7d0ec9009566fd6d3364a203d2c parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/b601081a-fee9-4865-b4b4-1d294eb8206d_isrForCrosstalkSources_996718_34.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/b601081a-fee9-4865-b4b4-1d294eb8206d_isrForCrosstalkSources_996718_34.stderr parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 6a5baf6456562db259fdfce386161e36 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7d1abd81-55ed-4415-80dc-9144f9c512ff_isrForCrosstalkSources_996716_11.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7d1abd81-55ed-4415-80dc-9144f9c512ff_isrForCrosstalkSources_996716_11.stderr parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 352b40aaedea52c50e34df0d50932d0e parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77911c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/b089c0ae-8e0c-4d2e-b8df-ab164abb522a_isrForCrosstalkSources_996719_32.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/b089c0ae-8e0c-4d2e-b8df-ab164abb522a_isrForCrosstalkSources_996719_32.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash a317b945c7aaa5df92bf9188e5df5b93 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f5a54abe-9e12-4862-9f19-3bdb267ee07a_isrForCrosstalkSources_996710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f5a54abe-9e12-4862-9f19-3bdb267ee07a_isrForCrosstalkSources_996710_43.stderr parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash cdb10b42a58842595e3e05d3ca0474f1 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1615b6b2-3c29-46dc-a9b1-64663c46ece5_isrForCrosstalkSources_996712_54.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1615b6b2-3c29-46dc-a9b1-64663c46ece5_isrForCrosstalkSources_996712_54.stderr parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 79d011369ec4f5658dfe97b3a9f21cac parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77913a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/94ca0ad9-b6a1-424f-8e3f-b7c9ab261134_isrForCrosstalkSources_996712_29.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/94ca0ad9-b6a1-424f-8e3f-b7c9ab261134_isrForCrosstalkSources_996712_29.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 4a7b292103b354f51df72c428fc55c41 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/cb40ae91-510a-4956-b4bd-056fcd9be8f3_isrForCrosstalkSources_996715_7.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/cb40ae91-510a-4956-b4bd-056fcd9be8f3_isrForCrosstalkSources_996715_7.stderr parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash bc2bdf970d15215c1022ae429c0cf2a6 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77914e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/785e5351-8a98-47ce-9e5e-2dbc2fdaad49_isrForCrosstalkSources_996717_5.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/785e5351-8a98-47ce-9e5e-2dbc2fdaad49_isrForCrosstalkSources_996717_5.stderr parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 2dcb12b6e123d90e651d6f06b4d8fd83 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/722c1973-cae8-45cd-9584-a3e9f346bd54_isrForCrosstalkSources_996711_35.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/722c1973-cae8-45cd-9584-a3e9f346bd54_isrForCrosstalkSources_996711_35.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 8672e2f631314d2e22d34492f25a9439 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ecba0ccf-1fb7-4f79-b597-e70d7baba7d6_isrForCrosstalkSources_996714_48.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ecba0ccf-1fb7-4f79-b597-e70d7baba7d6_isrForCrosstalkSources_996714_48.stderr parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 94219e936e18f98772a79877a9c2fccd parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77916c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8ee3a6a1-0371-4ef0-a2a8-43a208a9cc37_isrForCrosstalkSources_996711_47.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8ee3a6a1-0371-4ef0-a2a8-43a208a9cc37_isrForCrosstalkSources_996711_47.stderr parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 9db078b4b81f7f8cd4dcef6f91242d6c parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f06c94fe-4b10-4c9f-8039-d0c521af4ea8_isrForCrosstalkSources_996709_51.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f06c94fe-4b10-4c9f-8039-d0c521af4ea8_isrForCrosstalkSources_996709_51.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 3028f344abf5385d79dbd8f6411aa994 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ff7fb28a-1787-4100-b300-dd3a87ee4399_isrForCrosstalkSources_996717_52.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ff7fb28a-1787-4100-b300-dd3a87ee4399_isrForCrosstalkSources_996717_52.stderr parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 9463395332383404389f2b45b634151d parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/707f6d4a-4284-48af-b4c4-a1dcee552e49_isrForCrosstalkSources_996711_56.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/707f6d4a-4284-48af-b4c4-a1dcee552e49_isrForCrosstalkSources_996711_56.stderr parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 8bef247555db1d585ce53d0ad6540bcf parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77918a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f4827e38-3e83-44dc-97f9-978e0d1824c3_isrForCrosstalkSources_996713_52.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f4827e38-3e83-44dc-97f9-978e0d1824c3_isrForCrosstalkSources_996713_52.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 49443d767720fba26e090ddb6935ba8b parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77919e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f971576e-a8c2-4a52-8336-748d58b03916_isrForCrosstalkSources_996705_48.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f971576e-a8c2-4a52-8336-748d58b03916_isrForCrosstalkSources_996705_48.stderr parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 62d8d6336e06d8d6c783fb5a823e93eb parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a546f8d7-df09-4364-9ca9-1ccff58fb09b_isrForCrosstalkSources_996709_13.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a546f8d7-df09-4364-9ca9-1ccff58fb09b_isrForCrosstalkSources_996709_13.stderr parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash bbbd6a575d502324751d8459ada5f390 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c09f7128-73fe-4d56-bec4-47a9199dc104_isrForCrosstalkSources_996713_49.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c09f7128-73fe-4d56-bec4-47a9199dc104_isrForCrosstalkSources_996713_49.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 23e081fb5b62abe6415ecea05538eac5 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/629fbf0d-114c-495b-a5d4-7524581070f6_isrForCrosstalkSources_996712_15.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/629fbf0d-114c-495b-a5d4-7524581070f6_isrForCrosstalkSources_996712_15.stderr parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 6c3a8130abd462ba26a55849a0d58837 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1b750868-0d4c-4368-9c34-27084e35289e_isrForCrosstalkSources_996712_50.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1b750868-0d4c-4368-9c34-27084e35289e_isrForCrosstalkSources_996712_50.stderr parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash c0c884e3a168a30758d44f2d59040ea5 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6dbc93ac-7406-4607-a263-b95ef2715b99_isrForCrosstalkSources_996710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6dbc93ac-7406-4607-a263-b95ef2715b99_isrForCrosstalkSources_996710_1.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 6a7be501452d07e2f483dba0ecfe7169 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/c7e7f013-abd9-463a-b2ba-824ab3f151ef_isrForCrosstalkSources_996720_3.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/c7e7f013-abd9-463a-b2ba-824ab3f151ef_isrForCrosstalkSources_996720_3.stderr parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 6d93501997846f4bad4efcaebc5ec718 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/598c3750-7dc6-4ec5-963d-13f42cc21db3_isrForCrosstalkSources_996710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/598c3750-7dc6-4ec5-963d-13f42cc21db3_isrForCrosstalkSources_996710_57.stderr parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 47918cc0f18c96959fae94e22f56d580 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/9533f097-04ce-4c5f-9a96-b13cf3af78d6_isrForCrosstalkSources_996721_11.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/9533f097-04ce-4c5f-9a96-b13cf3af78d6_isrForCrosstalkSources_996721_11.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 276265d049905e3d271ec73bd3e2ad8f parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/caef9236-54d9-4861-8917-7cd4834340ba_isrForCrosstalkSources_996719_24.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/caef9236-54d9-4861-8917-7cd4834340ba_isrForCrosstalkSources_996719_24.stderr parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash e48021d85ab00667ba48196286dc6b03 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7791ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/48f81c04-d93a-41a2-9066-22ed728d0ed4_isrForCrosstalkSources_996719_35.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/48f81c04-d93a-41a2-9066-22ed728d0ed4_isrForCrosstalkSources_996719_35.stderr parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash be7ebb5ea838234d3e6a9670b5cfaaba parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/ef864896-801e-4e9d-a6ea-f192e0093ed2_isrForCrosstalkSources_996712_22.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/ef864896-801e-4e9d-a6ea-f192e0093ed2_isrForCrosstalkSources_996712_22.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash a13f620065e944ecaf8ae113e23d1e71 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77920c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/da2d356f-cd44-40f2-8f02-6bf84535ac2f_isrForCrosstalkSources_996715_9.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/da2d356f-cd44-40f2-8f02-6bf84535ac2f_isrForCrosstalkSources_996715_9.stderr parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash daa5b3f4c6b939e21be04de4062ea950 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3da5e83c-bbce-4e9d-995e-5af3f49bf702_isrForCrosstalkSources_996719_34.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3da5e83c-bbce-4e9d-995e-5af3f49bf702_isrForCrosstalkSources_996719_34.stderr parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 421c1ba728dea8c2a388e1bb85f34da9 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77922a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/42c197e8-25dc-4e16-bcd6-f353ac73ab39_isrForCrosstalkSources_996719_56.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/42c197e8-25dc-4e16-bcd6-f353ac73ab39_isrForCrosstalkSources_996719_56.stderr 2024-03-15 12:16:50 proc_lsst.multi:170 [INFO] [multi] job_id 17071130 proc_lsst.multi INFO: [multi] job_id 17071130 2024-03-15 12:16:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 12:16:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071130 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071130 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17071130 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-15 12:16:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-15 12:16:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 12:16:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash e5df5272eb1db9648bb92b12a67cea8d parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/e2d08e00-5def-42b3-afc9-3ebb36d8de28_isrForCrosstalkSources_996710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/e2d08e00-5def-42b3-afc9-3ebb36d8de28_isrForCrosstalkSources_996710_25.stderr parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 694d85479688e0cd00b35374c3e2b3e8 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/cdaf9858-65f1-4e6b-a949-489a57956073_isrForCrosstalkSources_996720_29.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/cdaf9858-65f1-4e6b-a949-489a57956073_isrForCrosstalkSources_996720_29.stderr parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash ba6391dda86bf543b2ad9ced7ed49f05 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/4e2772f4-2d6c-4ef3-9e7f-8cadc2918c55_isrForCrosstalkSources_996720_26.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/4e2772f4-2d6c-4ef3-9e7f-8cadc2918c55_isrForCrosstalkSources_996720_26.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 2279a713484369cb8968f1c1476c2e87 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/61fded40-6164-440a-a4d8-4cdce6f6eb99_isrForCrosstalkSources_996708_16.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/61fded40-6164-440a-a4d8-4cdce6f6eb99_isrForCrosstalkSources_996708_16.stderr parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash f62d9854ee7f0cf6b90dd818dc05cbd4 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/34acdd12-1c01-4b73-9a0e-4855d4d19060_isrForCrosstalkSources_996719_29.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/34acdd12-1c01-4b73-9a0e-4855d4d19060_isrForCrosstalkSources_996719_29.stderr parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 79b7417aa78a46c4c5f945fc8fd7f74e parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77923e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/d5470117-627b-4a5c-9e34-4c20435668f2_isrForCrosstalkSources_996718_33.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/d5470117-627b-4a5c-9e34-4c20435668f2_isrForCrosstalkSources_996718_33.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 20f1bb483418fc8a12aad082b7f13fe7 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ff377424-7822-45fa-8521-66f7b839b3fb_isrForCrosstalkSources_996705_46.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ff377424-7822-45fa-8521-66f7b839b3fb_isrForCrosstalkSources_996705_46.stderr parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 6f7a278314e061959bdf2e4936783b38 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/55986c27-819d-4ac7-bacc-c3b37ae4f5bc_isrForCrosstalkSources_996716_31.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/55986c27-819d-4ac7-bacc-c3b37ae4f5bc_isrForCrosstalkSources_996716_31.stderr parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 43f277ec17805c0341857ad0b24242fe parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ec28e420-e14e-40dc-88ce-b824014214f4_isrForCrosstalkSources_996714_37.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ec28e420-e14e-40dc-88ce-b824014214f4_isrForCrosstalkSources_996714_37.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash fd9554396687eabbe7b8d32cd1a7eadb parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/adba7068-9284-496e-ac82-124521ac607b_isrForCrosstalkSources_996716_2.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/adba7068-9284-496e-ac82-124521ac607b_isrForCrosstalkSources_996716_2.stderr parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 747ad59434c1d9ae45912a7fdb944457 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77925c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/6c841711-5ac3-42c6-b643-2fbe28e92dd2_isrForCrosstalkSources_996715_2.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/6c841711-5ac3-42c6-b643-2fbe28e92dd2_isrForCrosstalkSources_996715_2.stderr parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash d52c53b689f82d06592c0588fde3d2bb parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b1e46745-33c4-443f-90df-02564ee571fc_isrForCrosstalkSources_996716_40.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b1e46745-33c4-443f-90df-02564ee571fc_isrForCrosstalkSources_996716_40.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash d8b0bbe5f20fca336e6ff569d4b2afaa parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77928e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/8f87cbc8-6dfd-434a-a548-6845107a71ad_isrForCrosstalkSources_996707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/8f87cbc8-6dfd-434a-a548-6845107a71ad_isrForCrosstalkSources_996707_31.stderr parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 7bef3209d7cd57c4b0c5f6385e5f2e48 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c31a41fc-8cd8-4da6-8ca5-270953d54ec9_isrForCrosstalkSources_996706_50.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c31a41fc-8cd8-4da6-8ca5-270953d54ec9_isrForCrosstalkSources_996706_50.stderr parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 0cc4aa4c0b0a56d890787f6cab3d0a4d parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/7a91dacf-1670-45ee-8979-994ebe0d64a2_isrForCrosstalkSources_996718_48.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/7a91dacf-1670-45ee-8979-994ebe0d64a2_isrForCrosstalkSources_996718_48.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 981ae060202629c101f460065da7ca3d parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/43dad802-25f2-468b-bdb2-29a00b55c7c7_isrForCrosstalkSources_996709_11.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/43dad802-25f2-468b-bdb2-29a00b55c7c7_isrForCrosstalkSources_996709_11.stderr parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 2dc75a33089991b9960747fe0ca5becf parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77927a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/12d1b418-8010-475c-8c73-6498dc02514d_isrForCrosstalkSources_996719_42.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/12d1b418-8010-475c-8c73-6498dc02514d_isrForCrosstalkSources_996719_42.stderr parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash a19bd0fc078e78e4339c6574bef44b76 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a5115635-1809-41e4-94b7-f99dad91b4eb_isrForCrosstalkSources_996712_55.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a5115635-1809-41e4-94b7-f99dad91b4eb_isrForCrosstalkSources_996712_55.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 59555f5113ffdf1351fffdeb9f6474e2 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/6cc05ddc-5341-4540-8d10-5c62d908721a_isrForCrosstalkSources_996716_46.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/6cc05ddc-5341-4540-8d10-5c62d908721a_isrForCrosstalkSources_996716_46.stderr parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash cbe21c4145535df686b1cecfc54425f5 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8b2c77e4-87ff-4164-9332-30ad577ee8f3_isrForCrosstalkSources_996711_59.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8b2c77e4-87ff-4164-9332-30ad577ee8f3_isrForCrosstalkSources_996711_59.stderr parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash cf711161788138f4c681b55202fa415b parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77939c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3c0c7f6b-1c24-4b9a-843a-f65e6c719f58_isrForCrosstalkSources_996705_8.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3c0c7f6b-1c24-4b9a-843a-f65e6c719f58_isrForCrosstalkSources_996705_8.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 8f2d54ff1309c5a4b1bdf5846b70cfc0 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e6064a2e-0ae7-44da-9bf5-045e00f5d52f_isrForCrosstalkSources_996706_5.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e6064a2e-0ae7-44da-9bf5-045e00f5d52f_isrForCrosstalkSources_996706_5.stderr parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash b594c06a20b72a77238b154ad7fc5abf parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/340a6fdf-01b9-4fa0-a905-c29dd7c9df2f_isrForCrosstalkSources_996715_27.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/340a6fdf-01b9-4fa0-a905-c29dd7c9df2f_isrForCrosstalkSources_996715_27.stderr parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 654ad555baa8a92355b41134a3ef275e parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77937e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3a2ac72d-61f8-45a7-955d-95fedc6cc6f8_isrForCrosstalkSources_996711_11.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3a2ac72d-61f8-45a7-955d-95fedc6cc6f8_isrForCrosstalkSources_996711_11.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash b9f156280b5700c851111239efddeb24 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e0a509c6-6e24-497d-abbb-8b7a3cfbc01f_isrForCrosstalkSources_996716_53.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e0a509c6-6e24-497d-abbb-8b7a3cfbc01f_isrForCrosstalkSources_996716_53.stderr parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 2e522573995363be84d29f7eaa350e2a parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77936a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/40e46a1b-9278-4842-86f8-46acf1832a21_isrForCrosstalkSources_996706_52.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/40e46a1b-9278-4842-86f8-46acf1832a21_isrForCrosstalkSources_996706_52.stderr parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash c94c546d8c6f9061dab7f618288bb405 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4adfcf7d-65f1-4aa7-984b-c1fb529386f7_isrForCrosstalkSources_996707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4adfcf7d-65f1-4aa7-984b-c1fb529386f7_isrForCrosstalkSources_996707_45.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 66feeb8bda3553572878f4b2f47f7fe6 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/79ca0770-41e0-4747-a34c-523df2e1a1e9_isrForCrosstalkSources_996705_23.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/79ca0770-41e0-4747-a34c-523df2e1a1e9_isrForCrosstalkSources_996705_23.stderr parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 638d374ef1827d61c060693568b09a42 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a99e07c4-a151-491d-985d-a542e3a4a331_isrForCrosstalkSources_996709_23.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a99e07c4-a151-491d-985d-a542e3a4a331_isrForCrosstalkSources_996709_23.stderr parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash ffcee6ae9d3eec433cfeb4c0afa8269b parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d9b124be-7d93-4e74-adde-1a8b5a47986d_isrForCrosstalkSources_996713_15.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d9b124be-7d93-4e74-adde-1a8b5a47986d_isrForCrosstalkSources_996713_15.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 2fcb5614335f0e71b25e11e3d2aba6c7 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77934c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/95275c3d-8ede-4e3c-a7bb-ab64545c1243_isrForCrosstalkSources_996711_53.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/95275c3d-8ede-4e3c-a7bb-ab64545c1243_isrForCrosstalkSources_996711_53.stderr parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash e93e9e300ceea772175eda46b312c295 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/23c9b037-42e2-4a6c-bebc-7afe0c5358e1_isrForCrosstalkSources_996716_20.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/23c9b037-42e2-4a6c-bebc-7afe0c5358e1_isrForCrosstalkSources_996716_20.stderr parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash aa15d9a54cbb3274256a6e43cbbcb516 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a45e0698-b32c-4316-a48a-913b69ddc372_isrForCrosstalkSources_996706_4.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a45e0698-b32c-4316-a48a-913b69ddc372_isrForCrosstalkSources_996706_4.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 5e226c50582ab7580e78cdaaae42dcf6 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/b45e9b3c-c7fe-4c4f-9662-aebb1a1ec9a8_isrForCrosstalkSources_996710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/b45e9b3c-c7fe-4c4f-9662-aebb1a1ec9a8_isrForCrosstalkSources_996710_24.stderr parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash ea89c2345ee099d42bbb8c0413d3e700 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77932e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/954252f4-fc68-459c-9688-4d3fa375c749_isrForCrosstalkSources_996715_18.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/954252f4-fc68-459c-9688-4d3fa375c749_isrForCrosstalkSources_996715_18.stderr parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 8c40df5ff509965b978ad269b9fbfcfd parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77931a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/1fef5127-1668-41b9-8c48-30772b7a5681_isrForCrosstalkSources_996706_41.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/1fef5127-1668-41b9-8c48-30772b7a5681_isrForCrosstalkSources_996706_41.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash d8223144639821c143992ba78e5dcedc parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7793060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/32040880-c4a1-4c2b-b29d-598171b662eb_isrForCrosstalkSources_996716_29.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/32040880-c4a1-4c2b-b29d-598171b662eb_isrForCrosstalkSources_996716_29.stderr parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 0a875751753c7e94dc36e25911e21ad0 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5a30d2ac-7364-4403-bcc0-8baf75b49c24_isrForCrosstalkSources_996718_21.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5a30d2ac-7364-4403-bcc0-8baf75b49c24_isrForCrosstalkSources_996718_21.stderr parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 9db2b7b7ef80b4312efe69df1d04d3b9 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2a689c76-08d3-42e5-b5bd-258c9f873d23_isrForCrosstalkSources_996718_13.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2a689c76-08d3-42e5-b5bd-258c9f873d23_isrForCrosstalkSources_996718_13.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 3c5fb232ce7def9d714d21879812385a parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/398f926b-07e0-4eb0-abfe-367df68836c8_isrForCrosstalkSources_996712_45.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/398f926b-07e0-4eb0-abfe-367df68836c8_isrForCrosstalkSources_996712_45.stderr parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash ae155f61f14025524815daa079614a02 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a983dfda-47ec-4076-a970-299dc9a0c87e_isrForCrosstalkSources_996712_60.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a983dfda-47ec-4076-a970-299dc9a0c87e_isrForCrosstalkSources_996712_60.stderr parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash f427f3d9dc1bff0ed1152b61bcc39f37 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f5df80d5-1aee-44d0-b5ea-cb01e47c0d9b_isrForCrosstalkSources_996711_61.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f5df80d5-1aee-44d0-b5ea-cb01e47c0d9b_isrForCrosstalkSources_996711_61.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash d155ec5d8320de30a3240b346184a7e6 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/32dcc6d4-9148-4483-9021-d78604e81836_isrForCrosstalkSources_996715_40.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/32dcc6d4-9148-4483-9021-d78604e81836_isrForCrosstalkSources_996715_40.stderr parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 5375e889f48b3f48d24accd27e130db9 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/97dc6c80-3715-458c-8632-bee16f4c07d5_isrForCrosstalkSources_996711_46.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/97dc6c80-3715-458c-8632-bee16f4c07d5_isrForCrosstalkSources_996711_46.stderr parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 664012122c595f5420ace4d4dbbea9f0 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/0fea6995-4f61-46bb-a74f-5528445c5680_isrForCrosstalkSources_996716_35.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/0fea6995-4f61-46bb-a74f-5528445c5680_isrForCrosstalkSources_996716_35.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash c317f426bf804344447312fd8d8e4022 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/71a79d83-08a4-4c3c-a4e9-3094e68d5261_isrForCrosstalkSources_996707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/71a79d83-08a4-4c3c-a4e9-3094e68d5261_isrForCrosstalkSources_996707_39.stderr parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 17723714af736e22e2840433cabed5d6 parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:51 proc_lsst.multi:170 [INFO] [multi] job_id 17071131 proc_lsst.multi INFO: [multi] job_id 17071131 2024-03-15 12:16:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 12:16:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071131 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071131 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17071131 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-15 12:16:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-15 12:16:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 12:16:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2df6d598-251a-4edf-88e2-de029052cd5c_isrForCrosstalkSources_996714_10.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2df6d598-251a-4edf-88e2-de029052cd5c_isrForCrosstalkSources_996714_10.stderr parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 4a36d790d0eea4f777f1ff227e93fa2b parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508ccd9f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/ca50883d-fce0-4a33-882e-cb98e65cea95_isrForCrosstalkSources_996711_60.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/ca50883d-fce0-4a33-882e-cb98e65cea95_isrForCrosstalkSources_996711_60.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash a27bc1aec8d380f14a240d1e3384607b parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/80c2b2e8-cf6a-4e30-a712-eeb37f1bb2bd_isrForCrosstalkSources_996707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/80c2b2e8-cf6a-4e30-a712-eeb37f1bb2bd_isrForCrosstalkSources_996707_43.stderr parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 424168030e94cdc4955e5c5228aed538 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/7efa8a24-8b64-4988-a18b-c68eb101f000_isrForCrosstalkSources_996706_9.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/7efa8a24-8b64-4988-a18b-c68eb101f000_isrForCrosstalkSources_996706_9.stderr parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 53640ee03240a5a42e053f4cd974efa4 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/da740bd1-9912-410a-974e-b44bd8f3a3d5_isrForCrosstalkSources_996718_27.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/da740bd1-9912-410a-974e-b44bd8f3a3d5_isrForCrosstalkSources_996718_27.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 6ffde7ac798e704d8ef62109c9d8119c parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/ec41bd70-843b-44a9-bd09-27a59a475ee2_isrForCrosstalkSources_996721_60.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/ec41bd70-843b-44a9-bd09-27a59a475ee2_isrForCrosstalkSources_996721_60.stderr parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash fc174ef050bfc27ab3b6f47ef8317461 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8f225d1b-9659-4fda-a31f-5b62a6e43d73_isrForCrosstalkSources_996718_30.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8f225d1b-9659-4fda-a31f-5b62a6e43d73_isrForCrosstalkSources_996718_30.stderr parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 470843161ec98bb6ed3775ac22fee734 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/e2daca34-4c88-4e50-9306-ebf9022769b0_isrForCrosstalkSources_996705_16.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/e2daca34-4c88-4e50-9306-ebf9022769b0_isrForCrosstalkSources_996705_16.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 91bc4a852d118a4e2189edd14b8a3c2d parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/843fbbe4-b16a-40db-b778-499349baa574_isrForCrosstalkSources_996707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/843fbbe4-b16a-40db-b778-499349baa574_isrForCrosstalkSources_996707_21.stderr parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash e97cf3e2e4dd333a18fd7cec737d6e26 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a4a0de5d-2225-41d5-8ce1-2f3f988b959e_isrForCrosstalkSources_996720_36.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a4a0de5d-2225-41d5-8ce1-2f3f988b959e_isrForCrosstalkSources_996720_36.stderr parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash a300d9b783d293f9f93f8f240ab70ab2 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3f220ed7-ef9d-4a99-8d2d-4481be2ab796_isrForCrosstalkSources_996707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3f220ed7-ef9d-4a99-8d2d-4481be2ab796_isrForCrosstalkSources_996707_32.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 46ea4f78acdbaa2b18d610d91e168e5f parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/748ff9ff-33c3-47c5-a056-e42b7b9abe2b_isrForCrosstalkSources_996715_37.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/748ff9ff-33c3-47c5-a056-e42b7b9abe2b_isrForCrosstalkSources_996715_37.stderr parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 60507e205809ccde4aea4aac84b46ce3 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4092abab-3157-4659-a6d0-2885c85d8998_isrForCrosstalkSources_996713_60.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4092abab-3157-4659-a6d0-2885c85d8998_isrForCrosstalkSources_996713_60.stderr parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash b058de6261279133f1c95b3b54c6b58e parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9ec5b516-e06c-45e1-a717-8046efe65545_isrForCrosstalkSources_996706_57.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9ec5b516-e06c-45e1-a717-8046efe65545_isrForCrosstalkSources_996706_57.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 18f79b0ba0b446d898fd12dd9040b927 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f4f5b6b0-e06c-4fe9-9d51-b98c2301fcf4_isrForCrosstalkSources_996708_33.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f4f5b6b0-e06c-4fe9-9d51-b98c2301fcf4_isrForCrosstalkSources_996708_33.stderr parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 305e3c9bbd93fd4c2907a0bb6126c97d parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1ea9951a-a5c6-4fd1-aaa4-3543c253f5eb_isrForCrosstalkSources_996719_14.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1ea9951a-a5c6-4fd1-aaa4-3543c253f5eb_isrForCrosstalkSources_996719_14.stderr parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 88088904cc687fb0365325943f005402 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a62f6aee-6835-49cf-87bb-825ff2bdf5db_isrForCrosstalkSources_996712_47.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a62f6aee-6835-49cf-87bb-825ff2bdf5db_isrForCrosstalkSources_996712_47.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 515f4b1784abcf76797eeb4c9477b2ba parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c62d1590-104a-4192-bccf-ad1532741fb9_isrForCrosstalkSources_996708_59.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c62d1590-104a-4192-bccf-ad1532741fb9_isrForCrosstalkSources_996708_59.stderr parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash fa569bd50dac7d12b13635912ad4523a parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2f0d31fe-cb27-446a-9e5d-7ffed2892922_isrForCrosstalkSources_996718_8.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2f0d31fe-cb27-446a-9e5d-7ffed2892922_isrForCrosstalkSources_996718_8.stderr parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 8fea1721a3ae2b1d58e01c4d8abe4c6e parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ca051e81-f6e4-48cc-a187-3c7e1e9775ee_isrForCrosstalkSources_996706_47.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ca051e81-f6e4-48cc-a187-3c7e1e9775ee_isrForCrosstalkSources_996706_47.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 82fb6932534c221e4c85b51981a42e6d parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/2fc1f67d-b903-44b7-9b54-66631b30a6bd_isrForCrosstalkSources_996707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/2fc1f67d-b903-44b7-9b54-66631b30a6bd_isrForCrosstalkSources_996707_28.stderr parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 85ffc1e22364051a59b001ba7b15ca0c parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fe7a4716-6785-485b-b1e4-f19a5a86a16b_isrForCrosstalkSources_996705_38.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fe7a4716-6785-485b-b1e4-f19a5a86a16b_isrForCrosstalkSources_996705_38.stderr parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash c512ff3f833682e3b6585151423e73d1 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/331645e6-df66-4664-8ef5-5b075ee05958_isrForCrosstalkSources_996717_62.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/331645e6-df66-4664-8ef5-5b075ee05958_isrForCrosstalkSources_996717_62.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 4baf3a0716a6effeaa1c0584ad2de893 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d95fcbf3-a6b3-48f4-be4d-33889a047cac_isrForCrosstalkSources_996708_54.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d95fcbf3-a6b3-48f4-be4d-33889a047cac_isrForCrosstalkSources_996708_54.stderr parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash b730d0e694e8ffa66f28ceb5e8c93c31 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/db80551e-95b2-4a4c-b8e0-9bf1ad0191f1_isrForCrosstalkSources_996720_48.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/db80551e-95b2-4a4c-b8e0-9bf1ad0191f1_isrForCrosstalkSources_996720_48.stderr parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash a8f0be95d80f65e84ff02dad994da58f parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/7da022a5-8372-4eed-8785-de5b5448f506_isrForCrosstalkSources_996719_39.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/7da022a5-8372-4eed-8785-de5b5448f506_isrForCrosstalkSources_996719_39.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash b037ed8e57a49d39f804ddc58010365a parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/101f17e7-fb96-4550-86b7-2a3b1ed78209_isrForCrosstalkSources_996720_62.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/101f17e7-fb96-4550-86b7-2a3b1ed78209_isrForCrosstalkSources_996720_62.stderr parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash eb12e00495f5eb0106aa0ae62bea393c parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/00517c26-941a-49c6-b6fd-9cde9bdfce5e_isrForCrosstalkSources_996719_48.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/00517c26-941a-49c6-b6fd-9cde9bdfce5e_isrForCrosstalkSources_996719_48.stderr parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 6989ad076bba614f7d10fe37a9d703b6 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/5cbc30ba-0c4e-4683-8593-6efc4757e061_isrForCrosstalkSources_996716_44.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/5cbc30ba-0c4e-4683-8593-6efc4757e061_isrForCrosstalkSources_996716_44.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 175be7804519a3c0b90fb4a909a309db parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3a7d183b-39e0-4428-bbdc-c86c5b929b45_isrForCrosstalkSources_996706_3.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3a7d183b-39e0-4428-bbdc-c86c5b929b45_isrForCrosstalkSources_996706_3.stderr parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash d40278a15f9b76553a7b2c875fbf4bf2 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/0655c96a-0153-46a9-9c06-823457d0174d_isrForCrosstalkSources_996710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/0655c96a-0153-46a9-9c06-823457d0174d_isrForCrosstalkSources_996710_58.stderr parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 72b019c8737349e3a09f69b70db60a51 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1bf8001d-617c-4f1f-8dc5-16b6b91fe398_isrForCrosstalkSources_996717_48.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1bf8001d-617c-4f1f-8dc5-16b6b91fe398_isrForCrosstalkSources_996717_48.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash fef7eec4d4ee594dd4b3b3807d511e24 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/61ddc714-68c5-469f-9311-44191d673f8d_isrForCrosstalkSources_996716_7.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/61ddc714-68c5-469f-9311-44191d673f8d_isrForCrosstalkSources_996716_7.stderr parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 45b1b8fc00c39303ec7c5fee586c66ef parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cae750c5-ce93-4ad0-be34-53d7920d3dea_isrForCrosstalkSources_996714_60.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cae750c5-ce93-4ad0-be34-53d7920d3dea_isrForCrosstalkSources_996714_60.stderr parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 531da7d74f1a064d93d2e82e2a1f7266 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f23d6ca6-91c1-4db0-96a2-ebcb55e716e1_isrForCrosstalkSources_996711_2.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f23d6ca6-91c1-4db0-96a2-ebcb55e716e1_isrForCrosstalkSources_996711_2.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash c3e0ba119db30d5118a04edcb3b84f5f parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3be8ff12-9d5d-4419-b671-3d993b23ebc9_isrForCrosstalkSources_996711_27.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3be8ff12-9d5d-4419-b671-3d993b23ebc9_isrForCrosstalkSources_996711_27.stderr parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash fbb96d7dcdfaf742ac34ebdf293fe6c8 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/daeffda5-61e9-46a6-9e2c-9de7594a073b_isrForCrosstalkSources_996710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/daeffda5-61e9-46a6-9e2c-9de7594a073b_isrForCrosstalkSources_996710_53.stderr parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 403540ad6df03f3770c74b206b9eb20c parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/d059ecb2-aa8b-490c-bd40-e22ca6d69c0a_isrForCrosstalkSources_996715_51.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/d059ecb2-aa8b-490c-bd40-e22ca6d69c0a_isrForCrosstalkSources_996715_51.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 9a21de2bb3c55172de90185b1ba17831 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/cf0167b8-8a61-4320-afe4-58ef44f25b6c_isrForCrosstalkSources_996706_51.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/cf0167b8-8a61-4320-afe4-58ef44f25b6c_isrForCrosstalkSources_996706_51.stderr parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 0b37e2d02c157e1ba2bb430390cd9fa6 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/08daf021-340b-4efb-baf7-47881afa6cc4_isrForCrosstalkSources_996714_15.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/08daf021-340b-4efb-baf7-47881afa6cc4_isrForCrosstalkSources_996714_15.stderr parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 10f8c52548aadf824309868fc1da51b7 parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ee95b117-fc81-4f47-b7ac-0e14431f6a2d_isrForCrosstalkSources_996716_8.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ee95b117-fc81-4f47-b7ac-0e14431f6a2d_isrForCrosstalkSources_996716_8.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash 46e1370425878a2ef529135c19bcbecf parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ed182106-6e05-4902-ae81-45823fff1455_isrForCrosstalkSources_996705_42.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ed182106-6e05-4902-ae81-45823fff1455_isrForCrosstalkSources_996705_42.stderr parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 7b59db8b1dc32c711510163c9142093c parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/21ffcce3-00bd-4cb1-92ab-d7c13deac713_isrForCrosstalkSources_996717_23.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/21ffcce3-00bd-4cb1-92ab-d7c13deac713_isrForCrosstalkSources_996717_23.stderr parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash a6b7f1f7781048cb4c7b5436fa02a47d parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/a447d3fe-de3c-41e9-af10-9b6cb5439537_isrForCrosstalkSources_996721_29.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/a447d3fe-de3c-41e9-af10-9b6cb5439537_isrForCrosstalkSources_996721_29.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 710c151605abde9e3c728f11a3f1e97e parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a4415684-fab1-4c0a-be71-6cec140fec2f_isrForCrosstalkSources_996706_14.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a4415684-fab1-4c0a-be71-6cec140fec2f_isrForCrosstalkSources_996706_14.stderr parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash fd0d98ef63873aecdb8fdd212a47572a parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b7516859-d1ac-4992-ab06-1e8de9a5f7ef_isrForCrosstalkSources_996706_19.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b7516859-d1ac-4992-ab06-1e8de9a5f7ef_isrForCrosstalkSources_996706_19.stderr parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash bf7b713e3c9510de37ae8876f964525e parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/becc23f7-9a4f-474b-980e-8d12cfe8e22a_isrForCrosstalkSources_996711_41.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/becc23f7-9a4f-474b-980e-8d12cfe8e22a_isrForCrosstalkSources_996711_41.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 5bb69d59a1c240fce696b538b60e473d parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/af6abadf-b6a4-4a68-9b4b-5493d73b9149_isrForCrosstalkSources_996707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/af6abadf-b6a4-4a68-9b4b-5493d73b9149_isrForCrosstalkSources_996707_47.stderr parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash fbe70a4bfb5ea4a198af67f6a6bf863f parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:52 proc_lsst.multi:170 [INFO] [multi] job_id 17071132 proc_lsst.multi INFO: [multi] job_id 17071132 2024-03-15 12:16:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 12:16:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071132 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071132 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17071132 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-15 12:16:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-15 12:16:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 12:16:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1d615cda-67da-4064-ae9f-5432c37fa18d_isrForCrosstalkSources_996705_31.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1d615cda-67da-4064-ae9f-5432c37fa18d_isrForCrosstalkSources_996705_31.stderr parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash ffafa59ce3cafd0bd40b653dea8a0669 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e40dee66-074f-42e1-b1ff-24da304ab578_isrForCrosstalkSources_996713_46.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e40dee66-074f-42e1-b1ff-24da304ab578_isrForCrosstalkSources_996713_46.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash c397912cb0cab7a0fa68976bbbe30dd2 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e451b33f-93c0-4118-a93e-0c89d8e603c0_isrForCrosstalkSources_996713_28.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e451b33f-93c0-4118-a93e-0c89d8e603c0_isrForCrosstalkSources_996713_28.stderr parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 25b64b613eceb0522e99595ac366dda8 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dce0fcb2-3d7f-4037-bc26-c31c71529acc_isrForCrosstalkSources_996711_24.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dce0fcb2-3d7f-4037-bc26-c31c71529acc_isrForCrosstalkSources_996711_24.stderr parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 3af3069828f4cd6d3aea4f8b889e20bd parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/62901d6d-2e73-4cec-8ed3-5b4e6552cd45_isrForCrosstalkSources_996711_48.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/62901d6d-2e73-4cec-8ed3-5b4e6552cd45_isrForCrosstalkSources_996711_48.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 2cfcc3365822ced8ab2d0c90a4312a91 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f7bbafe2-622d-452d-adb6-23ff734a5b19_isrForCrosstalkSources_996708_58.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f7bbafe2-622d-452d-adb6-23ff734a5b19_isrForCrosstalkSources_996708_58.stderr parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash e685e157dd75c584ccb16f92af92e25d parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c1031d8f-7686-46a9-aa86-1a5ac6399f48_isrForCrosstalkSources_996714_14.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c1031d8f-7686-46a9-aa86-1a5ac6399f48_isrForCrosstalkSources_996714_14.stderr parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash fea93153fd568ed26ed7557cb6e5c9f9 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/96bb2524-e54d-4c3a-9d8b-f1b327f19ad4_isrForCrosstalkSources_996705_2.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/96bb2524-e54d-4c3a-9d8b-f1b327f19ad4_isrForCrosstalkSources_996705_2.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash dbe61f894a43a8c0fcf550adb082a769 parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/b8244049-3cdb-421b-a52b-5f121edbdbc4_isrForCrosstalkSources_996707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/b8244049-3cdb-421b-a52b-5f121edbdbc4_isrForCrosstalkSources_996707_9.stderr parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 0d825ea46d2d842f319025385cd56157 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/de266c85-c632-475b-b6e5-6c901354e1bb_isrForCrosstalkSources_996721_39.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/de266c85-c632-475b-b6e5-6c901354e1bb_isrForCrosstalkSources_996721_39.stderr parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 554b976b762cb38f94e04436b9be5a28 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/5b2480cc-04ed-41c5-94b8-408438896e8e_isrForCrosstalkSources_996711_54.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/5b2480cc-04ed-41c5-94b8-408438896e8e_isrForCrosstalkSources_996711_54.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 30114f917b295a9aa8304e71380dbb84 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/bcc7ff04-a3d5-4b26-95c0-d49766386b6f_isrForCrosstalkSources_996706_33.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/bcc7ff04-a3d5-4b26-95c0-d49766386b6f_isrForCrosstalkSources_996706_33.stderr parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 8f61b46954978cabf61876b1ad8c3ea8 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/a71f163d-cadf-4113-acac-b6d9e10e4c0d_isrForCrosstalkSources_996716_37.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/a71f163d-cadf-4113-acac-b6d9e10e4c0d_isrForCrosstalkSources_996716_37.stderr parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash e7f4379be60002e8427a59acce531925 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a6067d59-0659-491a-aae6-acda34114bd7_isrForCrosstalkSources_996720_13.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a6067d59-0659-491a-aae6-acda34114bd7_isrForCrosstalkSources_996720_13.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 58be5efbdd180050524ca64a5afcf538 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f780d3b1-5c25-4c95-9aec-89efcbe4bfed_isrForCrosstalkSources_996714_32.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f780d3b1-5c25-4c95-9aec-89efcbe4bfed_isrForCrosstalkSources_996714_32.stderr parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 173aa01fdfd73007ecca00b998a1a579 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c46ff2aa-d2aa-4d26-a932-04886f351b72_isrForCrosstalkSources_996714_22.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c46ff2aa-d2aa-4d26-a932-04886f351b72_isrForCrosstalkSources_996714_22.stderr parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 6fec45fc45a387cc3bc4b7f343c61c67 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fe8851e0-0df0-4f10-8ce1-b37f35602c18_isrForCrosstalkSources_996721_50.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fe8851e0-0df0-4f10-8ce1-b37f35602c18_isrForCrosstalkSources_996721_50.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 9b9ecfe90282227c12b89bd622f8bbd5 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5a02a25d-6231-49ee-b506-565af09c0555_isrForCrosstalkSources_996708_14.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5a02a25d-6231-49ee-b506-565af09c0555_isrForCrosstalkSources_996708_14.stderr parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash df74f22257bd7b97d0325e7deabd656d parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/fa9fad0f-ea6d-4faa-b3b1-e642bd687f52_isrForCrosstalkSources_996711_31.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/fa9fad0f-ea6d-4faa-b3b1-e642bd687f52_isrForCrosstalkSources_996711_31.stderr parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash d1ca85eb7fc8cd644ae3e549815093ff parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e33e8e70-522e-4785-b6d6-5c499b99f880_isrForCrosstalkSources_996708_12.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e33e8e70-522e-4785-b6d6-5c499b99f880_isrForCrosstalkSources_996708_12.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 4bb1e3facd6709494ad01b08d22aa1dd parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/56d11554-844e-4c7e-8646-bf0d893058e3_isrForCrosstalkSources_996706_20.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/56d11554-844e-4c7e-8646-bf0d893058e3_isrForCrosstalkSources_996706_20.stderr parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 3b1e5586ec003602e252893018afd1d6 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/655ace0a-b5d5-4849-8774-609f96321a88_isrForCrosstalkSources_996710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/655ace0a-b5d5-4849-8774-609f96321a88_isrForCrosstalkSources_996710_13.stderr parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash c77e61ba22bbe364278efe10e64f8514 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/d0f7d54c-819b-4a68-b01e-64e9413fc175_isrForCrosstalkSources_996720_5.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/d0f7d54c-819b-4a68-b01e-64e9413fc175_isrForCrosstalkSources_996720_5.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 7cc0038be8bc2ccb729f623d0adf05b0 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/391a39d8-4e27-4d6f-9bab-a4a60228d164_isrForCrosstalkSources_996716_60.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/391a39d8-4e27-4d6f-9bab-a4a60228d164_isrForCrosstalkSources_996716_60.stderr parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash ff973a09c6ee3f94e1441f3f65f0adc2 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7dbb9aed-fd83-4943-a451-08ffac6aaab2_isrForCrosstalkSources_996716_59.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7dbb9aed-fd83-4943-a451-08ffac6aaab2_isrForCrosstalkSources_996716_59.stderr parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash a18d5b987bde330dc42301d035a8f8e3 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4a904bbf-ad2a-4ff9-a25b-76e344433513_isrForCrosstalkSources_996719_25.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4a904bbf-ad2a-4ff9-a25b-76e344433513_isrForCrosstalkSources_996719_25.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash f44753281c73dfd94b3453371c74b751 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/49e8aee2-d99d-4237-9dfa-f73749a40324_isrForCrosstalkSources_996710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/49e8aee2-d99d-4237-9dfa-f73749a40324_isrForCrosstalkSources_996710_59.stderr parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash fffe76acc63835a2d149753638f00516 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/66f0f740-891e-491f-a30c-102c35933bde_isrForCrosstalkSources_996705_39.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/66f0f740-891e-491f-a30c-102c35933bde_isrForCrosstalkSources_996705_39.stderr parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash f6cb279e00a0208954c2597f61083f64 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/e11f5da9-1c09-409f-9712-6fe7121cc7c0_isrForCrosstalkSources_996718_61.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/e11f5da9-1c09-409f-9712-6fe7121cc7c0_isrForCrosstalkSources_996718_61.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash f2719054b9ccb5f21db6204fb1ad2a1c parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8d551041-da58-40e8-a455-6b2429ccfd8d_isrForCrosstalkSources_996715_59.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8d551041-da58-40e8-a455-6b2429ccfd8d_isrForCrosstalkSources_996715_59.stderr parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 85615e002a94e0443c7c0d387aa06594 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/42942d77-f864-4b2d-b7ff-a5951a720380_isrForCrosstalkSources_996716_15.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/42942d77-f864-4b2d-b7ff-a5951a720380_isrForCrosstalkSources_996716_15.stderr parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 9432b4b55c386b6cc1b3aae4f21a4500 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/aa819c81-0e73-4860-9dee-b10f8a38f1cf_isrForCrosstalkSources_996716_32.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/aa819c81-0e73-4860-9dee-b10f8a38f1cf_isrForCrosstalkSources_996716_32.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash abfd6a88b9eb91e8715128354d740683 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f315a9db-8395-476c-a734-8b5669168960_isrForCrosstalkSources_996713_3.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f315a9db-8395-476c-a734-8b5669168960_isrForCrosstalkSources_996713_3.stderr parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 82176d2d27818dab10b088f57b7dfefb parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5c899371-de50-461b-811b-689df1cb2c0f_isrForCrosstalkSources_996708_20.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5c899371-de50-461b-811b-689df1cb2c0f_isrForCrosstalkSources_996708_20.stderr parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 6ca0fb331828e7e1ff92b1d549818b51 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1867911f-08ec-4dfa-89c1-33e60a42a52a_isrForCrosstalkSources_996721_7.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1867911f-08ec-4dfa-89c1-33e60a42a52a_isrForCrosstalkSources_996721_7.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash b8580e201be00e169c6644687ae1ff42 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/b5ff4568-658c-4df0-9901-d50f977f5558_isrForCrosstalkSources_996711_18.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/b5ff4568-658c-4df0-9901-d50f977f5558_isrForCrosstalkSources_996711_18.stderr parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash b7c685881fae457a2b812c3fc6a2d36e parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9c226a6f-0f2b-41d9-a8c6-32607cb7f617_isrForCrosstalkSources_996708_24.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9c226a6f-0f2b-41d9-a8c6-32607cb7f617_isrForCrosstalkSources_996708_24.stderr parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash b0f7c4eadd07dc6881fc4fef0182e1f8 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5062b486-c032-4adb-a528-952f7cdf5e1c_isrForCrosstalkSources_996710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5062b486-c032-4adb-a528-952f7cdf5e1c_isrForCrosstalkSources_996710_33.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 5d8027c17a6f88867e325a3e733addef parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/6ca7a0e5-1cd6-4678-8cd7-6802b757692f_isrForCrosstalkSources_996717_46.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/6ca7a0e5-1cd6-4678-8cd7-6802b757692f_isrForCrosstalkSources_996717_46.stderr parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash e89dc5617570ead19fc2c3f8e49e1746 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/0b19c764-26a7-4b70-8a99-ef0c9b93b988_isrForCrosstalkSources_996721_6.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/0b19c764-26a7-4b70-8a99-ef0c9b93b988_isrForCrosstalkSources_996721_6.stderr parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 9f3c92ee3666f0c8a7cc74e995b5e2e6 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/a17acf34-d6ad-4660-8f26-a795d5fb09e6_isrForCrosstalkSources_996711_40.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/a17acf34-d6ad-4660-8f26-a795d5fb09e6_isrForCrosstalkSources_996711_40.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 80cd2121714d436dee5d5e660f03f663 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b6c7429f-2021-43a8-aee5-6aa03d6e47a0_isrForCrosstalkSources_996716_12.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b6c7429f-2021-43a8-aee5-6aa03d6e47a0_isrForCrosstalkSources_996716_12.stderr parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 4e1b685abf88dfef0ec7d5adc4d99f35 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e86b85e6-6595-4b2d-b9d6-a594a794436c_isrForCrosstalkSources_996709_62.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e86b85e6-6595-4b2d-b9d6-a594a794436c_isrForCrosstalkSources_996709_62.stderr parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash b7c0e25ca0a2714506c75ffd04001ff3 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/608af338-da1e-45e5-bc6e-3993ce2675d5_isrForCrosstalkSources_996706_61.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/608af338-da1e-45e5-bc6e-3993ce2675d5_isrForCrosstalkSources_996706_61.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 8a4d75d66fa3da07cd93c8b60965d97e parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/7c7e742a-65fb-4ab1-9782-d1a9c1e00f71_isrForCrosstalkSources_996713_40.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/7c7e742a-65fb-4ab1-9782-d1a9c1e00f71_isrForCrosstalkSources_996713_40.stderr parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 8d2fd77e33f5c1e378dd4243301cff37 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:52 proc_lsst.multi:170 [INFO] [multi] job_id 17071133 proc_lsst.multi INFO: [multi] job_id 17071133 2024-03-15 12:16:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 12:16:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071133 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071133 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17071133 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-15 12:16:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-15 12:16:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 12:16:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cb893e40-c24c-4925-93e5-806961273bbd_isrForCrosstalkSources_996714_16.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cb893e40-c24c-4925-93e5-806961273bbd_isrForCrosstalkSources_996714_16.stderr parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash c9dcf33b13b8e70e6ede8294efce298d parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7792480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8829d55a-cfa9-498a-8e80-5c9a890bd8a7_isrForCrosstalkSources_996721_32.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8829d55a-cfa9-498a-8e80-5c9a890bd8a7_isrForCrosstalkSources_996721_32.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 91f0112d0b7dedd0d8f47c987c580c34 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fe6bd841-3667-4a2a-a0b2-391b37eb3b89_isrForCrosstalkSources_996717_1.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fe6bd841-3667-4a2a-a0b2-391b37eb3b89_isrForCrosstalkSources_996717_1.stderr parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 3fab2d979a96ecf4a149d189a7685755 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/7cb3d603-5a26-409f-b2f3-9e335b387804_isrForCrosstalkSources_996707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/7cb3d603-5a26-409f-b2f3-9e335b387804_isrForCrosstalkSources_996707_40.stderr parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash ff360b10907dac5a8ae5e8a6d28ea773 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/2bd8210c-4601-4d78-9e4b-2cf39b68f954_isrForCrosstalkSources_996715_55.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/2bd8210c-4601-4d78-9e4b-2cf39b68f954_isrForCrosstalkSources_996715_55.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 183390c7485fefb1bad92412731fb9fb parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/261b534e-080e-4857-9595-f51ada08b25e_isrForCrosstalkSources_996710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/261b534e-080e-4857-9595-f51ada08b25e_isrForCrosstalkSources_996710_61.stderr parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 591d431927dbc944eaf61e8238d8979c parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/63798b98-8742-4e78-9c64-309e1196e94a_isrForCrosstalkSources_996706_8.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/63798b98-8742-4e78-9c64-309e1196e94a_isrForCrosstalkSources_996706_8.stderr parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 38a9a1c91d9600071085a8fdc14d8f3f parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/aeb11c8d-8c99-4bdb-8a05-59f628ba9621_isrForCrosstalkSources_996707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/aeb11c8d-8c99-4bdb-8a05-59f628ba9621_isrForCrosstalkSources_996707_50.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 2e468d796f2be67777d7f7680905d696 parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/2ba1b89d-f171-4627-b18e-580e79c163e3_isrForCrosstalkSources_996721_5.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/2ba1b89d-f171-4627-b18e-580e79c163e3_isrForCrosstalkSources_996721_5.stderr parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash a8b6927fe870d5b486b25ffaa4913f76 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5e10c774-8eee-448c-963e-ec79e4c308b0_isrForCrosstalkSources_996710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5e10c774-8eee-448c-963e-ec79e4c308b0_isrForCrosstalkSources_996710_28.stderr parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 41ff49c7961ed395bbde518ac52e6567 parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/56e0a820-a5d9-420d-913f-24b4b5078cab_isrForCrosstalkSources_996718_6.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/56e0a820-a5d9-420d-913f-24b4b5078cab_isrForCrosstalkSources_996718_6.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 2881c8bb42d822588152d6891deb85b2 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/a7057946-401a-44a5-a28f-ff139ad50fd6_isrForCrosstalkSources_996711_16.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/a7057946-401a-44a5-a28f-ff139ad50fd6_isrForCrosstalkSources_996711_16.stderr parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash bdac53c24fac1cfccc2d672a6f2869c1 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8d3db3b8-d11a-4a6e-a5eb-2091f2796997_isrForCrosstalkSources_996706_7.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8d3db3b8-d11a-4a6e-a5eb-2091f2796997_isrForCrosstalkSources_996706_7.stderr parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash d23b4605c87eef77d706c3b5676b463a parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/02bf03cd-9eec-409c-a3e8-fa333983b847_isrForCrosstalkSources_996721_3.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/02bf03cd-9eec-409c-a3e8-fa333983b847_isrForCrosstalkSources_996721_3.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 2588086f63604e631b37b07058509d85 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4a527693-5782-449e-abfb-e5b30c37482e_isrForCrosstalkSources_996707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4a527693-5782-449e-abfb-e5b30c37482e_isrForCrosstalkSources_996707_11.stderr parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 4bbb3ec7e883284375ffeb567348a5c1 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/3bb66f9e-bda1-41a3-8afd-609613acc80a_isrForCrosstalkSources_996721_41.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/3bb66f9e-bda1-41a3-8afd-609613acc80a_isrForCrosstalkSources_996721_41.stderr parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 93eeb7544909e04a0c358594b7829aa4 parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/8ccbf1a9-f72b-41b3-b98d-c42519948f78_isrForCrosstalkSources_996719_54.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/8ccbf1a9-f72b-41b3-b98d-c42519948f78_isrForCrosstalkSources_996719_54.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 727cdad7b2457d339bda68888a51f989 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/9c404c15-4c68-4893-980c-6dc930732078_isrForCrosstalkSources_996711_42.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/9c404c15-4c68-4893-980c-6dc930732078_isrForCrosstalkSources_996711_42.stderr parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 880e8f6422ea4ed8fea599f4100ac50f parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a2e874af-caa1-48e7-b9cd-5bf59811cddb_isrForCrosstalkSources_996713_41.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a2e874af-caa1-48e7-b9cd-5bf59811cddb_isrForCrosstalkSources_996713_41.stderr parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 8fb27ef2636a80bcaa7f68ae074fe189 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e4ca6888-4904-49c5-93dd-33e4793bcede_isrForCrosstalkSources_996707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e4ca6888-4904-49c5-93dd-33e4793bcede_isrForCrosstalkSources_996707_56.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash b462ec7c3d80ce3ac30f7b7e36d043ac parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3df6e00f-44e7-4c93-9069-6fdb877fb898_isrForCrosstalkSources_996719_45.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3df6e00f-44e7-4c93-9069-6fdb877fb898_isrForCrosstalkSources_996719_45.stderr parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 83ca41af3b693e7053021f87391cee69 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/1194829d-3506-4d55-98e7-70d6fd31da97_isrForCrosstalkSources_996716_43.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/1194829d-3506-4d55-98e7-70d6fd31da97_isrForCrosstalkSources_996716_43.stderr parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 9a77553d33f7dee2d953d89b0f33c033 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ad1d3177-b410-46a3-afc0-6862010ae003_isrForCrosstalkSources_996707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ad1d3177-b410-46a3-afc0-6862010ae003_isrForCrosstalkSources_996707_14.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash ca6e73f2808ab75c06f63c5fc4f70d83 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a82ae4d5-84ee-458b-8e08-0026df8c28c8_isrForCrosstalkSources_996719_41.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a82ae4d5-84ee-458b-8e08-0026df8c28c8_isrForCrosstalkSources_996719_41.stderr parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash b36b001061edcb2f283f3003f728b2b8 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/fd5f43fa-0642-41e1-93a4-9aaa745b7e05_isrForCrosstalkSources_996713_9.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/fd5f43fa-0642-41e1-93a4-9aaa745b7e05_isrForCrosstalkSources_996713_9.stderr parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash ac977ec258cddcd4140887d29799bcef parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ddcb3d4b-5805-4306-89d1-27f179c324e9_isrForCrosstalkSources_996714_44.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ddcb3d4b-5805-4306-89d1-27f179c324e9_isrForCrosstalkSources_996714_44.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash eee6b31fa4dd0992732badb48bf65296 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c7885d89-d166-402e-b0ec-84e77432abb0_isrForCrosstalkSources_996705_14.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c7885d89-d166-402e-b0ec-84e77432abb0_isrForCrosstalkSources_996705_14.stderr parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash b49b052277f5b53179f84a7543e4feac parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/a8de82b9-e96d-4b87-ad9e-ed7ad1f2d0f2_isrForCrosstalkSources_996716_25.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/a8de82b9-e96d-4b87-ad9e-ed7ad1f2d0f2_isrForCrosstalkSources_996716_25.stderr parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 8abe5d703bed0c9597c5de42d49ae70e parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/14f33a52-c732-4dc7-bcd3-799243443f17_isrForCrosstalkSources_996720_56.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/14f33a52-c732-4dc7-bcd3-799243443f17_isrForCrosstalkSources_996720_56.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash b08e4c8ad019f1c74ba31aeeffebb4a2 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/d915f432-9b36-4ab6-88bc-b8942e44ff72_isrForCrosstalkSources_996710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/d915f432-9b36-4ab6-88bc-b8942e44ff72_isrForCrosstalkSources_996710_8.stderr parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 6061ccd6e505c940136d41f1521894a2 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8c500436-d09d-4f48-9eed-1d06767452f5_isrForCrosstalkSources_996706_42.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8c500436-d09d-4f48-9eed-1d06767452f5_isrForCrosstalkSources_996706_42.stderr parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 44f0e2903e52c62f5cc7f37069ceec00 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7cb3a98a-4882-4a72-bf42-2e4b7f0d39a9_isrForCrosstalkSources_996715_12.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7cb3a98a-4882-4a72-bf42-2e4b7f0d39a9_isrForCrosstalkSources_996715_12.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 6c604c7794a678fcc2c46e5ae9cc743a parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/48967121-50d2-460f-b70a-617677a88cd0_isrForCrosstalkSources_996706_28.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/48967121-50d2-460f-b70a-617677a88cd0_isrForCrosstalkSources_996706_28.stderr parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 1f2955b33edd822d508db1224c2e45e6 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f4164667-3c89-40e0-8911-d47321716e1e_isrForCrosstalkSources_996716_27.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f4164667-3c89-40e0-8911-d47321716e1e_isrForCrosstalkSources_996716_27.stderr parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 57c13c7045e37a78035ef7e677b8c3d4 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7baa5323-6499-46c8-8aaa-1d9d6ade1eff_isrForCrosstalkSources_996715_44.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7baa5323-6499-46c8-8aaa-1d9d6ade1eff_isrForCrosstalkSources_996715_44.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 46ad87e86ca8b503cf0f2bf78dd60ce4 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b458d01f-00c0-4bf7-a67f-32d57e6fb57a_isrForCrosstalkSources_996713_36.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b458d01f-00c0-4bf7-a67f-32d57e6fb57a_isrForCrosstalkSources_996713_36.stderr parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash d53236c89e1509024310746e041459ec parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/22423cd8-57de-4dda-899e-74e5514f468c_isrForCrosstalkSources_996716_52.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/22423cd8-57de-4dda-899e-74e5514f468c_isrForCrosstalkSources_996716_52.stderr parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 19087d215e747acc625b6f824122fe82 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/3a201cb0-38d5-46c1-84bf-bc66719c0750_isrForCrosstalkSources_996718_28.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/3a201cb0-38d5-46c1-84bf-bc66719c0750_isrForCrosstalkSources_996718_28.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 19390cf5e929156eb228d35b9614d8bf parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ff3bc849-2ccb-435b-8330-880e05123253_isrForCrosstalkSources_996720_60.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ff3bc849-2ccb-435b-8330-880e05123253_isrForCrosstalkSources_996720_60.stderr parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 9a6513ed573db7fea9d66b156d8ccaab parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/976aa0e6-1384-49b4-b5f0-2730e333a7a4_isrForCrosstalkSources_996719_38.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/976aa0e6-1384-49b4-b5f0-2730e333a7a4_isrForCrosstalkSources_996719_38.stderr parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash dd9efc1edfe6e55ad43c42c2f6d2b5aa parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/408f693c-187c-4631-b3a4-2ea643d733d3_isrForCrosstalkSources_996708_6.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/408f693c-187c-4631-b3a4-2ea643d733d3_isrForCrosstalkSources_996708_6.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash aca3c8ded18db2bd3eab99e0f893725f parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/d5bc9797-a566-45a5-9275-c5b767dbe80d_isrForCrosstalkSources_996718_23.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/d5bc9797-a566-45a5-9275-c5b767dbe80d_isrForCrosstalkSources_996718_23.stderr parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 0a9451483c211fec436e51fa5f51f4ad parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/4a505199-cb02-40a7-b14c-d44a1747b022_isrForCrosstalkSources_996710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/4a505199-cb02-40a7-b14c-d44a1747b022_isrForCrosstalkSources_996710_36.stderr parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 281022bc9dd72f9b24fe95e24345b613 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6883b5b3-52e5-4389-849b-7b5bcb35990c_isrForCrosstalkSources_996710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6883b5b3-52e5-4389-849b-7b5bcb35990c_isrForCrosstalkSources_996710_52.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 69e311c9613e3dee0378e29a8faf5f11 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ac29fd84-4a75-4d90-a232-501dd1cfa8c5_isrForCrosstalkSources_996706_46.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ac29fd84-4a75-4d90-a232-501dd1cfa8c5_isrForCrosstalkSources_996706_46.stderr parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 23390c506877e5faae489816e15221ee parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ecfb1e7f-302d-4023-acd5-5c4061a11c21_isrForCrosstalkSources_996706_59.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ecfb1e7f-302d-4023-acd5-5c4061a11c21_isrForCrosstalkSources_996706_59.stderr parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 2948a29c6b5a18112fe0151468ae6091 parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/ca7ebcc0-8aba-4b03-9148-f1f249d74d2b_isrForCrosstalkSources_996715_13.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/ca7ebcc0-8aba-4b03-9148-f1f249d74d2b_isrForCrosstalkSources_996715_13.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 0f07a633ab9c0bd0276e8c3a1e47fbab parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/bed6980b-1674-4e5e-b9d3-ceb023555256_isrForCrosstalkSources_996707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/bed6980b-1674-4e5e-b9d3-ceb023555256_isrForCrosstalkSources_996707_22.stderr parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 2c8e359aad2c6c0be6300bc3402a6692 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/660ca067-3bde-4c6d-8f84-e20b92cad1f7_isrForCrosstalkSources_996707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/660ca067-3bde-4c6d-8f84-e20b92cad1f7_isrForCrosstalkSources_996707_3.stderr parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash aa42bd0e7ccdcfd077f37162f97b516c parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/e6e9ae94-dc6c-4105-af3f-75c743fb3515_isrForCrosstalkSources_996718_22.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/e6e9ae94-dc6c-4105-af3f-75c743fb3515_isrForCrosstalkSources_996718_22.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash be48e837d23118dc0779f63548053590 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/fd283128-18b1-46de-b273-ead6e2983e4a_isrForCrosstalkSources_996716_13.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/fd283128-18b1-46de-b273-ead6e2983e4a_isrForCrosstalkSources_996716_13.stderr parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 5ddc5f7a2175ac8ce95e1f0cf030b0da parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/16499b23-2ef0-4a01-adbe-56f4ba19a56b_isrForCrosstalkSources_996713_33.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/16499b23-2ef0-4a01-adbe-56f4ba19a56b_isrForCrosstalkSources_996713_33.stderr parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 7eddb4ae70f1a4930e48b1ba298fccfc parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/6a84d1cb-8796-4ec9-af51-75f342c9e716_isrForCrosstalkSources_996715_47.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/6a84d1cb-8796-4ec9-af51-75f342c9e716_isrForCrosstalkSources_996715_47.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 839faadabf15e446c65b099a9b156c1b parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/470ede73-ff08-49d9-82d4-24ebb917cea1_isrForCrosstalkSources_996718_31.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/470ede73-ff08-49d9-82d4-24ebb917cea1_isrForCrosstalkSources_996718_31.stderr parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 4960618dc1e0911dc6c291542b7ce9e9 parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5f8ed275-68a8-4d9e-89fe-11a8f5cda06e_isrForCrosstalkSources_996707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5f8ed275-68a8-4d9e-89fe-11a8f5cda06e_isrForCrosstalkSources_996707_15.stderr parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash e29d01910b692e526704290c4c24991a parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:53 proc_lsst.multi:170 [INFO] [multi] job_id 17071134 proc_lsst.multi INFO: [multi] job_id 17071134 2024-03-15 12:16:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 12:16:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071134 parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1bc3a504-e82b-471c-831b-a5d50f9b9499_isrForCrosstalkSources_996717_30.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1bc3a504-e82b-471c-831b-a5d50f9b9499_isrForCrosstalkSources_996717_30.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 72a8ae6e65fb3ccbd2706cf6b5b63fbf parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071134 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17071134 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-15 12:16:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-15 12:16:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 12:16:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d5c8b7e0-73ba-4e22-b326-07bd95205439_isrForCrosstalkSources_996709_12.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d5c8b7e0-73ba-4e22-b326-07bd95205439_isrForCrosstalkSources_996709_12.stderr parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash d41400970e6309525291842ac114c35f parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7790c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/fc342a5c-4d6d-4c23-a8f8-2fb27e6c6c29_isrForCrosstalkSources_996715_19.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/fc342a5c-4d6d-4c23-a8f8-2fb27e6c6c29_isrForCrosstalkSources_996715_19.stderr parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash ca2c70c27ab6c1f9255628bfeb3d064d parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/24e83d4d-9b4e-4e48-bcb7-30ea7bd3eb38_isrForCrosstalkSources_996711_13.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/24e83d4d-9b4e-4e48-bcb7-30ea7bd3eb38_isrForCrosstalkSources_996711_13.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 2d3d56bb5f7c0c7cfa14969e58e7dc3b parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c77d7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6871ebeb-39f5-4ee9-842d-7335410eae21_isrForCrosstalkSources_996713_22.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6871ebeb-39f5-4ee9-842d-7335410eae21_isrForCrosstalkSources_996713_22.stderr parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 926aa91e9e63ba5378d7ac32a5076b23 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/cc333598-ddc4-456e-aed5-3a25174a21df_isrForCrosstalkSources_996705_34.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/cc333598-ddc4-456e-aed5-3a25174a21df_isrForCrosstalkSources_996705_34.stderr parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 162ba29ff5e39852336389329c811e94 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c773a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/f739bd2d-520a-4566-9c61-ac94914de218_isrForCrosstalkSources_996721_25.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/f739bd2d-520a-4566-9c61-ac94914de218_isrForCrosstalkSources_996721_25.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash ca0affed09335662b19cd3e66b7a1e8d parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/30571f27-f020-4fdc-89f2-daacc384ae9a_isrForCrosstalkSources_996710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/30571f27-f020-4fdc-89f2-daacc384ae9a_isrForCrosstalkSources_996710_47.stderr parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 4991949864f2996255ffe4950b5f7fe7 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7674180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f656146b-2d57-4fcc-b8c2-6417cef5b6e9_isrForCrosstalkSources_996711_25.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f656146b-2d57-4fcc-b8c2-6417cef5b6e9_isrForCrosstalkSources_996711_25.stderr parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 05c2d98ea66fefc3181e34d5a71bf8c9 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d89285be-3c54-4bae-9907-ef9c89ee5fe0_isrForCrosstalkSources_996709_54.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d89285be-3c54-4bae-9907-ef9c89ee5fe0_isrForCrosstalkSources_996709_54.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash e21ef05c1cc789456a0a48ed3bb9cd92 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8bd57c42-063a-4a61-89ec-9b6857af725f_isrForCrosstalkSources_996706_2.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8bd57c42-063a-4a61-89ec-9b6857af725f_isrForCrosstalkSources_996706_2.stderr parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash a0d0f45c86f5cb46751ee4d2838bd82a parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e98f5307-509d-4b2c-8773-e617d79f33f1_isrForCrosstalkSources_996720_23.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e98f5307-509d-4b2c-8773-e617d79f33f1_isrForCrosstalkSources_996720_23.stderr parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 4fa4658520704b5f0f71ee8e4d66de09 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/9b3a1979-a70a-4754-883d-76d6dbdc42c5_isrForCrosstalkSources_996716_1.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/9b3a1979-a70a-4754-883d-76d6dbdc42c5_isrForCrosstalkSources_996716_1.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 0f59e618a33a36a72e94eda13256bffe parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/d37f6fef-22c7-499f-9938-b9d2f834e060_isrForCrosstalkSources_996716_24.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/d37f6fef-22c7-499f-9938-b9d2f834e060_isrForCrosstalkSources_996716_24.stderr parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 2bc5a5d6d3981fd9cbfa1e1ab08d099c parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/7ec36e98-1df4-4de2-a384-bbe1ef850662_isrForCrosstalkSources_996709_32.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/7ec36e98-1df4-4de2-a384-bbe1ef850662_isrForCrosstalkSources_996709_32.stderr parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash f4bafe3ba1ccadb55396b3422132346a parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/cb307ce3-6d54-4d46-8c8d-366b8d12c331_isrForCrosstalkSources_996709_47.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/cb307ce3-6d54-4d46-8c8d-366b8d12c331_isrForCrosstalkSources_996709_47.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 2cd97ffaf8b6d6547920e0638a9a5897 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6249fc3e-4e15-495c-881b-5ee044e7b29d_isrForCrosstalkSources_996711_33.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6249fc3e-4e15-495c-881b-5ee044e7b29d_isrForCrosstalkSources_996711_33.stderr parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 3871454b2145abb343a708cd553ad58e parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e82ba63a-3ddb-469c-a98e-b5404d94987c_isrForCrosstalkSources_996707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e82ba63a-3ddb-469c-a98e-b5404d94987c_isrForCrosstalkSources_996707_41.stderr parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 058f8a1d4c63cb1870d9c63ab500dac5 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/9d9488f6-fdc5-4541-a56d-aa58fa6b0b67_isrForCrosstalkSources_996712_44.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/9d9488f6-fdc5-4541-a56d-aa58fa6b0b67_isrForCrosstalkSources_996712_44.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 8987ad4bcd3d6d01751a6eac3eb88f51 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/0983fa0c-cba3-4468-86af-520d9703c258_isrForCrosstalkSources_996707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/0983fa0c-cba3-4468-86af-520d9703c258_isrForCrosstalkSources_996707_23.stderr parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash cef88d5a88abd42400912cd8a35bebfb parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f05f2f59-8b02-430a-b40e-10969107d510_isrForCrosstalkSources_996712_35.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f05f2f59-8b02-430a-b40e-10969107d510_isrForCrosstalkSources_996712_35.stderr parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash d64c393dcf53e413055746c94ba6a765 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/d8bdc16c-e3ca-490d-94f5-18520e0d7cff_isrForCrosstalkSources_996707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/d8bdc16c-e3ca-490d-94f5-18520e0d7cff_isrForCrosstalkSources_996707_26.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash a07e5ff933411aa42961ce1cd8aad14d parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/465ae9b2-5b6f-4e4d-8562-557ad76c8b26_isrForCrosstalkSources_996713_27.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/465ae9b2-5b6f-4e4d-8562-557ad76c8b26_isrForCrosstalkSources_996713_27.stderr parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash b0bf744c40b0f9cfc972667ce1f30318 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/4b3924e4-0c7e-4980-b380-05e9b08e598d_isrForCrosstalkSources_996705_36.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/4b3924e4-0c7e-4980-b380-05e9b08e598d_isrForCrosstalkSources_996705_36.stderr parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash db6b7b56a7cdef0ba3e988b38cd8b7a1 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/82c69b8c-008d-4daf-9e66-1b2c89aa1808_isrForCrosstalkSources_996706_39.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/82c69b8c-008d-4daf-9e66-1b2c89aa1808_isrForCrosstalkSources_996706_39.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 1ecfd03f463376522ac6f9065d4ef49e parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/5bb9150c-87d4-47a5-999e-5304d45c3867_isrForCrosstalkSources_996715_31.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/5bb9150c-87d4-47a5-999e-5304d45c3867_isrForCrosstalkSources_996715_31.stderr parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 5552182579d5c1b08a64a5545baf7061 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/36cf8e2b-b8e5-4e3a-96e4-6f3aaa9c23c9_isrForCrosstalkSources_996721_59.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/36cf8e2b-b8e5-4e3a-96e4-6f3aaa9c23c9_isrForCrosstalkSources_996721_59.stderr parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash b54709b35241309db96e136e2ca6f4bc parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7e4c5b52-796e-491b-b3cd-23764137c4b0_isrForCrosstalkSources_996712_31.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7e4c5b52-796e-491b-b3cd-23764137c4b0_isrForCrosstalkSources_996712_31.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 3e4c92cf9e29711df13817b919d0fafa parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f9e01117-56c2-4f39-aa31-6b345110aba3_isrForCrosstalkSources_996711_5.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f9e01117-56c2-4f39-aa31-6b345110aba3_isrForCrosstalkSources_996711_5.stderr parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 3ec90da055fec061ece965c5f951ea9b parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/2148b870-693a-4d7b-8306-940ccb44c1cb_isrForCrosstalkSources_996716_47.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/2148b870-693a-4d7b-8306-940ccb44c1cb_isrForCrosstalkSources_996716_47.stderr parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 7dfbebe9bb7cc51a4001d64bc6f5a593 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fe08c529-e189-4bbb-8e23-56b85a244f98_isrForCrosstalkSources_996717_13.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fe08c529-e189-4bbb-8e23-56b85a244f98_isrForCrosstalkSources_996717_13.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 402a38e5da4734d1212b5a08771d701c parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8e7615f8-b2a1-414b-aa92-ca6a1c1b3334_isrForCrosstalkSources_996717_25.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8e7615f8-b2a1-414b-aa92-ca6a1c1b3334_isrForCrosstalkSources_996717_25.stderr parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash e8802693bca05d7a90c056f8d5036eea parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ea193423-59d8-4e8f-8654-5756a2ec50ff_isrForCrosstalkSources_996714_35.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ea193423-59d8-4e8f-8654-5756a2ec50ff_isrForCrosstalkSources_996714_35.stderr parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash e0b04165621e0f5507a551ec327ea9bf parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/12b62d9b-64f8-4d71-96ee-c794e00e1226_isrForCrosstalkSources_996712_4.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/12b62d9b-64f8-4d71-96ee-c794e00e1226_isrForCrosstalkSources_996712_4.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 0a675ac660debc7907c8bf33c6a595eb parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/3d050409-6b2c-4468-bb66-7867b40e9a67_isrForCrosstalkSources_996717_61.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/3d050409-6b2c-4468-bb66-7867b40e9a67_isrForCrosstalkSources_996717_61.stderr parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 305a58da318975e27691c9b4e6a245b8 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/7d914812-ab8f-4e52-99e6-8b1cb19d8309_isrForCrosstalkSources_996708_25.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/7d914812-ab8f-4e52-99e6-8b1cb19d8309_isrForCrosstalkSources_996708_25.stderr parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 49c270b862db2c8e8bd49fc7308fec82 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c08724f0-c5c5-4dd5-9446-e464767cf6e1_isrForCrosstalkSources_996719_36.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c08724f0-c5c5-4dd5-9446-e464767cf6e1_isrForCrosstalkSources_996719_36.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 7758c14761caffcc18ae032d574bac8b parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/37e8ca3e-bfae-4e5e-946a-2f76ea5f2abd_isrForCrosstalkSources_996708_26.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/37e8ca3e-bfae-4e5e-946a-2f76ea5f2abd_isrForCrosstalkSources_996708_26.stderr parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash f44982a670e32a82547110aea817662a parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76760c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4b9e0e84-8ece-4be5-a11d-c62c237db681_isrForCrosstalkSources_996714_33.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4b9e0e84-8ece-4be5-a11d-c62c237db681_isrForCrosstalkSources_996714_33.stderr parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 6a5f0383328faa1f7fb2c3fc33109f4c parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7675f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/c69250c3-7777-44b0-b28b-700d5320e007_isrForCrosstalkSources_996710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/c69250c3-7777-44b0-b28b-700d5320e007_isrForCrosstalkSources_996710_29.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 3a050c98d97a1634f465cba25aa6c1d3 parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76774c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f66d1cd7-63c2-45ff-bf9b-8e39801cc587_isrForCrosstalkSources_996716_26.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f66d1cd7-63c2-45ff-bf9b-8e39801cc587_isrForCrosstalkSources_996716_26.stderr parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 238c44bafa16cc460953b18e04f39cd8 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/42027b16-8f9b-44dc-881e-40194b05a6cc_isrForCrosstalkSources_996712_32.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/42027b16-8f9b-44dc-881e-40194b05a6cc_isrForCrosstalkSources_996712_32.stderr parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash c3c2e10e0324b423a8d4a7241b516b09 parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c6212a84-4a84-43fa-962b-aa6137468dc0_isrForCrosstalkSources_996713_34.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c6212a84-4a84-43fa-962b-aa6137468dc0_isrForCrosstalkSources_996713_34.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 7e38be7df33b6f614a686c5a85cc2096 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ae220146-660e-4852-9e8c-64b98d0f6b06_isrForCrosstalkSources_996709_36.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ae220146-660e-4852-9e8c-64b98d0f6b06_isrForCrosstalkSources_996709_36.stderr parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 775252fb2e62b475ac34c3ed741ab3f9 parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b865dd2c-f1b6-467c-904a-cad32cdba1b5_isrForCrosstalkSources_996715_50.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b865dd2c-f1b6-467c-904a-cad32cdba1b5_isrForCrosstalkSources_996715_50.stderr parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 4d9777f132e65ddb0fafc785110630d2 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3dc8824f-db04-44a7-9ca0-aaf6ee0d207d_isrForCrosstalkSources_996711_14.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3dc8824f-db04-44a7-9ca0-aaf6ee0d207d_isrForCrosstalkSources_996711_14.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash c862d8d3ebb07822a7d370d3d39da20e parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/33fb22fc-5b5b-46c6-9b8c-2025109bd786_isrForCrosstalkSources_996710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/33fb22fc-5b5b-46c6-9b8c-2025109bd786_isrForCrosstalkSources_996710_55.stderr parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 239942ba5006095fef36f6423ac4347b parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/b814e9f7-1723-4e07-9fdd-c8599b4f3d61_isrForCrosstalkSources_996709_39.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/b814e9f7-1723-4e07-9fdd-c8599b4f3d61_isrForCrosstalkSources_996709_39.stderr parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash b359aa77178d7db71163e8f67e0263c2 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/81f7a664-81e4-4921-9835-e7b66c71a4a0_isrForCrosstalkSources_996717_32.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/81f7a664-81e4-4921-9835-e7b66c71a4a0_isrForCrosstalkSources_996717_32.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 8430ca809fe698f5e830c77389920905 parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a4da0604-06b8-418d-bbe6-f5331f885138_isrForCrosstalkSources_996719_12.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a4da0604-06b8-418d-bbe6-f5331f885138_isrForCrosstalkSources_996719_12.stderr parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 8493a9be69e0da6f31f9fb44c3ecb35e parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/74b81d51-62cb-45e5-9b9d-fe32eec62686_isrForCrosstalkSources_996715_28.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/74b81d51-62cb-45e5-9b9d-fe32eec62686_isrForCrosstalkSources_996715_28.stderr parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash b1a512aee1ed3b62d1df7aeb588b81c6 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a5180114-a82d-4da4-a52a-4db6e6a28084_isrForCrosstalkSources_996720_34.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a5180114-a82d-4da4-a52a-4db6e6a28084_isrForCrosstalkSources_996720_34.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash a20c12bf107a71f7d7b58ceb5de47072 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f2ae9040-bccb-4cff-9095-8bdb43ec659a_isrForCrosstalkSources_996707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f2ae9040-bccb-4cff-9095-8bdb43ec659a_isrForCrosstalkSources_996707_51.stderr parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash dd4bc17e4e964878df29ecd7d1badfe3 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/683f0112-4abd-4eeb-a265-43becdeef65d_isrForCrosstalkSources_996708_7.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/683f0112-4abd-4eeb-a265-43becdeef65d_isrForCrosstalkSources_996708_7.stderr parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 9e7572c87f0ded1c645bcc4512d31329 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a87110db-ea0d-4798-b25a-83f220cf50e7_isrForCrosstalkSources_996720_8.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a87110db-ea0d-4798-b25a-83f220cf50e7_isrForCrosstalkSources_996720_8.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 37b899e0ed31fe57d825e570a9060605 parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/d7cd9c3b-7309-44c6-9dda-e7cdc7be2f6d_isrForCrosstalkSources_996720_45.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/d7cd9c3b-7309-44c6-9dda-e7cdc7be2f6d_isrForCrosstalkSources_996720_45.stderr parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash cc99f9613f305a07658f1e1383ed69c2 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/9dfbe05f-950d-448e-8ece-c14fc761213c_isrForCrosstalkSources_996716_14.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/9dfbe05f-950d-448e-8ece-c14fc761213c_isrForCrosstalkSources_996716_14.stderr parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 78110e3ff334c58e314d694c3934827f parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/963a0b81-41dc-45e2-9a2a-b46256dbe6a0_isrForCrosstalkSources_996720_54.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/963a0b81-41dc-45e2-9a2a-b46256dbe6a0_isrForCrosstalkSources_996720_54.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash b6157c85afeb873d7fae3d8d08b531c3 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/66025669-b784-4b4a-877e-d75cbd64e36e_isrForCrosstalkSources_996719_53.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/66025669-b784-4b4a-877e-d75cbd64e36e_isrForCrosstalkSources_996719_53.stderr parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash d0d256d5f908452a6bee5b192728f9a3 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/a1e39d73-7cf2-4499-ae4f-98faa14790e7_isrForCrosstalkSources_996710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/a1e39d73-7cf2-4499-ae4f-98faa14790e7_isrForCrosstalkSources_996710_17.stderr parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 3aebbabc6289a9ae47c32c2016a17b9b parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d59f39e1-c362-40ae-bc57-f3a030cde833_isrForCrosstalkSources_996711_37.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d59f39e1-c362-40ae-bc57-f3a030cde833_isrForCrosstalkSources_996711_37.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 790ba7f9f4fae431bd44c28c3517ffe2 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f14c3509-de8a-417a-b031-5b99d1fa09ce_isrForCrosstalkSources_996717_45.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f14c3509-de8a-417a-b031-5b99d1fa09ce_isrForCrosstalkSources_996717_45.stderr parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash bd5857bb28bb804c8cf19e78c10d11fd parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/42857f6c-66f2-43c6-aecd-f9ac08dde79a_isrForCrosstalkSources_996706_48.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/42857f6c-66f2-43c6-aecd-f9ac08dde79a_isrForCrosstalkSources_996706_48.stderr parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 0b725de2bfc572aeb80a1dcfa0ef564e parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/27a097e2-c283-4d40-98f7-c13d5e41ebd7_isrForCrosstalkSources_996706_53.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/27a097e2-c283-4d40-98f7-c13d5e41ebd7_isrForCrosstalkSources_996706_53.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 48dcc35a2bb2fd70f1d40b02aa5df085 parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3a6cc474-ea2f-412c-ba45-e900eeae8472_isrForCrosstalkSources_996707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3a6cc474-ea2f-412c-ba45-e900eeae8472_isrForCrosstalkSources_996707_29.stderr parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash a8e20b79a19136d5055e881272c560cf parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/94aafeb3-ec15-48fd-b144-8f105be80dd7_isrForCrosstalkSources_996718_57.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/94aafeb3-ec15-48fd-b144-8f105be80dd7_isrForCrosstalkSources_996718_57.stderr parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 9f3963c5ecc7c733e106e1393deb807a parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2523ec02-aa85-499c-982d-d11ad50f48fb_isrForCrosstalkSources_996713_50.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2523ec02-aa85-499c-982d-d11ad50f48fb_isrForCrosstalkSources_996713_50.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash 5f913a50844b25414c16abd77ed5f060 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/f24817c5-1af4-4a75-82a4-d0461e739a01_isrForCrosstalkSources_996720_40.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/f24817c5-1af4-4a75-82a4-d0461e739a01_isrForCrosstalkSources_996720_40.stderr parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash 26cc722e48d908d3dfaccfcf60830067 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5c29ba51-1dcd-4203-adda-a2a11f5e75ba_isrForCrosstalkSources_996719_1.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5c29ba51-1dcd-4203-adda-a2a11f5e75ba_isrForCrosstalkSources_996719_1.stderr parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 06b7babf40f87781fc3030ea1d045bb7 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/44a5b13c-48b7-49ba-96d8-6cab3934480c_isrForCrosstalkSources_996706_62.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/44a5b13c-48b7-49ba-96d8-6cab3934480c_isrForCrosstalkSources_996706_62.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 36196f3a2a65ebd67347488df30e3c1c parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/2a584321-eff6-41c2-8df7-89974dba5fd3_isrForCrosstalkSources_996706_36.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/2a584321-eff6-41c2-8df7-89974dba5fd3_isrForCrosstalkSources_996706_36.stderr parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash 9982fb18021d241afc7db43bcef0a7ab parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/baf53551-ea9b-4a09-bffd-36271e11e577_isrForCrosstalkSources_996710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/baf53551-ea9b-4a09-bffd-36271e11e577_isrForCrosstalkSources_996710_37.stderr parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash d307234de52b2ae39c2461f9d348192e parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ca30dd63-d485-4256-9d6a-1ffe7f09130d_isrForCrosstalkSources_996718_26.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ca30dd63-d485-4256-9d6a-1ffe7f09130d_isrForCrosstalkSources_996718_26.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 4940a2f75be22427448d151a12df9c75 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3b38afba-8498-4684-b855-01f1a6d5d9c0_isrForCrosstalkSources_996719_4.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3b38afba-8498-4684-b855-01f1a6d5d9c0_isrForCrosstalkSources_996719_4.stderr parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash a3f9e9eac0d14e32f0118f19f9c5c584 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/3e2552cc-9f33-4cff-9e14-eb2f8bd975ea_isrForCrosstalkSources_996712_5.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/3e2552cc-9f33-4cff-9e14-eb2f8bd975ea_isrForCrosstalkSources_996712_5.stderr parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 7627b0870ac0e22391ae44824ac946c7 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/b718d40b-087c-4216-b1aa-fd8aa0e51d18_isrForCrosstalkSources_996717_12.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/b718d40b-087c-4216-b1aa-fd8aa0e51d18_isrForCrosstalkSources_996717_12.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash c111a0d7b195accb5a19aff01e502735 parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/b4b50147-311e-42cd-9f3b-4d14f6f13f63_isrForCrosstalkSources_996714_13.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/b4b50147-311e-42cd-9f3b-4d14f6f13f63_isrForCrosstalkSources_996714_13.stderr parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash c96b2ab9c78464e407a95276e1567e13 parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/939ba26e-e659-4b2b-8406-543e3596047f_isrForCrosstalkSources_996714_11.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/939ba26e-e659-4b2b-8406-543e3596047f_isrForCrosstalkSources_996714_11.stderr parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 625abb7716f0cebc292ddc59c0819f21 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/3d87c29e-5cf8-4f89-85a6-27c0127ab9a7_isrForCrosstalkSources_996721_57.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/3d87c29e-5cf8-4f89-85a6-27c0127ab9a7_isrForCrosstalkSources_996721_57.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash d1b1d427354e58e026ef6b59710fa5b3 parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/5c9daf33-d02b-49be-970b-7972268347f9_isrForCrosstalkSources_996705_29.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/5c9daf33-d02b-49be-970b-7972268347f9_isrForCrosstalkSources_996705_29.stderr parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash abea6aa712196f43f86bf66095257eac parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/7a4176c3-c63d-4b1e-a085-fc92378db4f2_isrForCrosstalkSources_996718_18.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/7a4176c3-c63d-4b1e-a085-fc92378db4f2_isrForCrosstalkSources_996718_18.stderr parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 3efcc460f831da09ff33a02317341941 parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:54 proc_lsst.multi:170 [INFO] [multi] job_id 17071135 proc_lsst.multi INFO: [multi] job_id 17071135 2024-03-15 12:16:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 12:16:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071135 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071135 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17071135 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-15 12:16:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-15 12:16:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 12:16:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/e0c95001-7db4-49b2-a616-f4e6c362b039_isrForCrosstalkSources_996719_44.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/e0c95001-7db4-49b2-a616-f4e6c362b039_isrForCrosstalkSources_996719_44.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash cc03bfc2283be649aca5a40f5b125c11 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/9b2cbc86-b058-4bc0-904c-0001327b9fdc_isrForCrosstalkSources_996720_38.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/9b2cbc86-b058-4bc0-904c-0001327b9fdc_isrForCrosstalkSources_996720_38.stderr parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 22db113c9766b513d7de7d5fd5852623 parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6e702b3a-5717-40bb-94f7-bb8034129010_isrForCrosstalkSources_996710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6e702b3a-5717-40bb-94f7-bb8034129010_isrForCrosstalkSources_996710_20.stderr parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash e031032bb955d078994df683274f1856 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a691ce94-e350-4f9a-a73b-3067a5dd590b_isrForCrosstalkSources_996705_26.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a691ce94-e350-4f9a-a73b-3067a5dd590b_isrForCrosstalkSources_996705_26.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash cee660d14418cb438a0347f78a4f5e9f parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/d64a4867-eaf4-4807-8895-2ad7f6b55d95_isrForCrosstalkSources_996716_57.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/d64a4867-eaf4-4807-8895-2ad7f6b55d95_isrForCrosstalkSources_996716_57.stderr parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 790d23b0a6d04883521f1f549bf36fc4 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/584a0d2a-b578-41e9-9451-b3ce15e666aa_isrForCrosstalkSources_996707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/584a0d2a-b578-41e9-9451-b3ce15e666aa_isrForCrosstalkSources_996707_16.stderr parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 06ecfac5c7d577b3d8f92c250d2d232b parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/1dc29edd-0c13-436e-b9f4-5fb6cdab8290_isrForCrosstalkSources_996720_43.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/1dc29edd-0c13-436e-b9f4-5fb6cdab8290_isrForCrosstalkSources_996720_43.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 4330b750752ace925aca9640efe49598 parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c60b6fc5-1833-4522-ace4-67de7a4ba4c8_isrForCrosstalkSources_996712_30.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c60b6fc5-1833-4522-ace4-67de7a4ba4c8_isrForCrosstalkSources_996712_30.stderr parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 157f5f2b170cec22372a18f1ef9baa68 parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76ddb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/87e67bc2-8a0d-43eb-b067-7a62180ede8f_isrForCrosstalkSources_996713_1.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/87e67bc2-8a0d-43eb-b067-7a62180ede8f_isrForCrosstalkSources_996713_1.stderr parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash f4e9889b16fecd52577488c4e1341f08 parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/834cdccf-6c93-4dc7-a9b8-92e4c6e5afb1_isrForCrosstalkSources_996706_32.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/834cdccf-6c93-4dc7-a9b8-92e4c6e5afb1_isrForCrosstalkSources_996706_32.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 855ebaf974744ccb0933dfcfcb62e757 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76ddbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1b8f528f-206f-4d0a-b4d1-ec297a1075b3_isrForCrosstalkSources_996705_10.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1b8f528f-206f-4d0a-b4d1-ec297a1075b3_isrForCrosstalkSources_996705_10.stderr parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 552f425a36c2182aa88731dc76c04683 parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/a5cb819c-6378-46cc-86ac-b9e13cf3d788_isrForCrosstalkSources_996721_52.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/a5cb819c-6378-46cc-86ac-b9e13cf3d788_isrForCrosstalkSources_996721_52.stderr parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash aa648e53c54f96f04e2c8b0e29a18f27 parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76ddc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c51a8264-5506-41a8-9494-598e896160a6_isrForCrosstalkSources_996709_5.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c51a8264-5506-41a8-9494-598e896160a6_isrForCrosstalkSources_996709_5.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 1efe0393ecb048776dabb900823408ae parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76ddda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/bc1e5849-3b5e-4c27-a6df-b6828e2be4b1_isrForCrosstalkSources_996720_25.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/bc1e5849-3b5e-4c27-a6df-b6828e2be4b1_isrForCrosstalkSources_996720_25.stderr parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 393769e925eb1f831d3ddc6c5464f5dc parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76ddd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5cc9c406-ff35-42a6-9d2e-658d6fc0c8d2_isrForCrosstalkSources_996710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5cc9c406-ff35-42a6-9d2e-658d6fc0c8d2_isrForCrosstalkSources_996710_35.stderr parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash cb3e8b7851f9b6b6eb0ab50f9202953d parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76ddee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/272f7c1a-3230-4786-9811-8ff52175b9e1_isrForCrosstalkSources_996709_55.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/272f7c1a-3230-4786-9811-8ff52175b9e1_isrForCrosstalkSources_996709_55.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash d3b12e5851a4a22625041850b3241d7f parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76ddf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7052e061-53b6-4de3-b859-f3d7fe4f36e5_isrForCrosstalkSources_996716_5.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7052e061-53b6-4de3-b859-f3d7fe4f36e5_isrForCrosstalkSources_996716_5.stderr parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash 750b08a214cc4ed341724c557e66b986 parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e58bfe34-4749-46df-b0c3-79852b1126ef_isrForCrosstalkSources_996717_24.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e58bfe34-4749-46df-b0c3-79852b1126ef_isrForCrosstalkSources_996717_24.stderr parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 4f855249953ebe076aa6a222ad5f8265 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/18bd4882-35b0-4c85-a73a-87bc61f021f3_isrForCrosstalkSources_996713_21.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/18bd4882-35b0-4c85-a73a-87bc61f021f3_isrForCrosstalkSources_996713_21.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash b96bc0d8485d50368c388af90d359e39 parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/32ec9c26-783b-42bc-8465-112d0c915838_isrForCrosstalkSources_996709_1.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/32ec9c26-783b-42bc-8465-112d0c915838_isrForCrosstalkSources_996709_1.stderr parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash f833e027aa3d0f7752eff519637ba397 parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a28c8baa-4e10-4393-a8be-0b00cc6be964_isrForCrosstalkSources_996705_33.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a28c8baa-4e10-4393-a8be-0b00cc6be964_isrForCrosstalkSources_996705_33.stderr parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash fb7a447031ea088cd157b223c5cacc95 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e5e91fc7-810c-4cab-86fc-7031bd744896_isrForCrosstalkSources_996709_56.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e5e91fc7-810c-4cab-86fc-7031bd744896_isrForCrosstalkSources_996709_56.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 913567e98786fd1b0d27702304ba7305 parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2ceac0a4-93d6-4791-8cdd-b36779bd7996_isrForCrosstalkSources_996710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2ceac0a4-93d6-4791-8cdd-b36779bd7996_isrForCrosstalkSources_996710_11.stderr parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 989101baf2707aacf3b80eee10937e26 parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/eb722317-9f88-471d-89ef-5aeb79b7c506_isrForCrosstalkSources_996708_39.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/eb722317-9f88-471d-89ef-5aeb79b7c506_isrForCrosstalkSources_996708_39.stderr parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash f579cbb9e58a3a77f320ed41b83af34b parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4e9d5ad9-5508-45f2-9446-e52c7a48f685_isrForCrosstalkSources_996707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4e9d5ad9-5508-45f2-9446-e52c7a48f685_isrForCrosstalkSources_996707_27.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash 089e9b3488d72db3692bf065e447f9e3 parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/1628241e-5feb-454b-b12f-8abe1dac1bce_isrForCrosstalkSources_996715_62.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/1628241e-5feb-454b-b12f-8abe1dac1bce_isrForCrosstalkSources_996715_62.stderr parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash dfd069be8041ee1e22ba7ee0be38224d parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b02c220f-f7d5-4f5d-9aac-7bc075c6b319_isrForCrosstalkSources_996713_8.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b02c220f-f7d5-4f5d-9aac-7bc075c6b319_isrForCrosstalkSources_996713_8.stderr parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 3143d30f6ad5c81432058af4ff5d4a8e parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/77c3154f-5c06-453c-b751-359ad2dc2434_isrForCrosstalkSources_996714_46.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/77c3154f-5c06-453c-b751-359ad2dc2434_isrForCrosstalkSources_996714_46.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 3464b36e615176125fc3301e67d5cbea parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/3d0d4505-e752-4519-80d8-45b1999dfd60_isrForCrosstalkSources_996714_61.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/3d0d4505-e752-4519-80d8-45b1999dfd60_isrForCrosstalkSources_996714_61.stderr parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 1ae9b66c3f8a146e9025614d1e7455d3 parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2faff2e1-06c3-41e0-bcdc-0b9d4fc7872c_isrForCrosstalkSources_996718_14.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2faff2e1-06c3-41e0-bcdc-0b9d4fc7872c_isrForCrosstalkSources_996718_14.stderr parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash ff113a0bb7f5a64590aa2f8710d3cfd2 parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7539b546-5023-4df1-a385-21b303350e96_isrForCrosstalkSources_996715_32.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7539b546-5023-4df1-a385-21b303350e96_isrForCrosstalkSources_996715_32.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 5d1e68cbed8b98d4f07be2b211b761be parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7ab68d7a-58d6-43a7-b08f-6aa9df1b28e6_isrForCrosstalkSources_996715_10.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7ab68d7a-58d6-43a7-b08f-6aa9df1b28e6_isrForCrosstalkSources_996715_10.stderr parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash b14d8d0ef16c467ecf8d2abba4fba2c8 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c29c46e6-e215-4bdd-b857-9b94dfe3e725_isrForCrosstalkSources_996714_56.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c29c46e6-e215-4bdd-b857-9b94dfe3e725_isrForCrosstalkSources_996714_56.stderr parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash 8639789ff64d7c8ae2eeec904f437dd1 parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76deac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/39a495ca-a749-45fa-aec7-e6795f463e60_isrForCrosstalkSources_996705_60.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/39a495ca-a749-45fa-aec7-e6795f463e60_isrForCrosstalkSources_996705_60.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 2f30a2ed2c9cad45e448ff11d6fd5c10 parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76de8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/aa3d520a-f3eb-4ea5-8e5a-5f207f97e0ee_isrForCrosstalkSources_996708_28.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/aa3d520a-f3eb-4ea5-8e5a-5f207f97e0ee_isrForCrosstalkSources_996708_28.stderr parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 84ee4740345771a598dd1d817d242146 parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2066d0fc-d081-423e-8503-325ca900a448_isrForCrosstalkSources_996712_48.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2066d0fc-d081-423e-8503-325ca900a448_isrForCrosstalkSources_996712_48.stderr parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash d4eb2ea683b2c856fbcc78a75d9ba63c parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/97bec4ae-97e9-40bf-95f5-1bf40ac30025_isrForCrosstalkSources_996712_24.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/97bec4ae-97e9-40bf-95f5-1bf40ac30025_isrForCrosstalkSources_996712_24.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 9376947155bcb865950b7855ac700f7e parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76deca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/8c56dd30-9eb1-40c5-81ab-9a2d293a97f9_isrForCrosstalkSources_996709_14.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/8c56dd30-9eb1-40c5-81ab-9a2d293a97f9_isrForCrosstalkSources_996709_14.stderr parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 7046891d2332e7ac1a65efa692af023a parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c93b7a70-bb95-4139-ba0b-bae8fa39544c_isrForCrosstalkSources_996712_10.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c93b7a70-bb95-4139-ba0b-bae8fa39544c_isrForCrosstalkSources_996712_10.stderr parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash bbba4cf637d6e91025f07ccba3422636 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76deb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/511eae03-067a-40bd-8677-da3f8db8504f_isrForCrosstalkSources_996710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/511eae03-067a-40bd-8677-da3f8db8504f_isrForCrosstalkSources_996710_49.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash d2c6d563422b378213a4e3bc46b36f9d parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/cf4221ac-ecb0-41f4-993b-71a5f384fe7a_isrForCrosstalkSources_996712_46.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/cf4221ac-ecb0-41f4-993b-71a5f384fe7a_isrForCrosstalkSources_996712_46.stderr parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash c9ae595dc0fecdc4f37c635f3a93c770 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76def20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/2a9fa2e7-f555-4ad0-af09-583ed06a3fc4_isrForCrosstalkSources_996719_40.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/2a9fa2e7-f555-4ad0-af09-583ed06a3fc4_isrForCrosstalkSources_996719_40.stderr parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash e92659536f3047daac88bfbff6f137ca parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76ded40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/8abf1a8b-97f4-48ab-85ee-e570c9e220d6_isrForCrosstalkSources_996719_7.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/8abf1a8b-97f4-48ab-85ee-e570c9e220d6_isrForCrosstalkSources_996719_7.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 5f49c3e51ea0603c3fdfd4cdbc7f54e2 parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/610ef7c2-26c6-432c-8c4c-33a16c3d04db_isrForCrosstalkSources_996705_58.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/610ef7c2-26c6-432c-8c4c-33a16c3d04db_isrForCrosstalkSources_996705_58.stderr parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash c9a27a021d1bab4014aaf98a605f6d45 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ddccbf67-7269-43a3-8850-4c1ae0892107_isrForCrosstalkSources_996713_2.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ddccbf67-7269-43a3-8850-4c1ae0892107_isrForCrosstalkSources_996713_2.stderr parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash dbeab97a133e0e122fec5cfb7386d109 parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/4365776c-45fd-4f4b-af82-174527b443ed_isrForCrosstalkSources_996712_13.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/4365776c-45fd-4f4b-af82-174527b443ed_isrForCrosstalkSources_996712_13.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 89a04d155ca1f72c6aa931ae75f132d4 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76defc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/86b56294-3f1a-4f3b-876a-dbef9771e45f_isrForCrosstalkSources_996711_34.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/86b56294-3f1a-4f3b-876a-dbef9771e45f_isrForCrosstalkSources_996711_34.stderr parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash 78f686d77999cd14a9358ef34230ebb1 parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cbafee00-b884-4c85-a1a5-035c879951ff_isrForCrosstalkSources_996710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cbafee00-b884-4c85-a1a5-035c879951ff_isrForCrosstalkSources_996710_12.stderr parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash 3c44c70c3641c10683ae2b9c45353ed2 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/673a0b39-8e7f-4551-857f-bf39b7fc73ba_isrForCrosstalkSources_996710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/673a0b39-8e7f-4551-857f-bf39b7fc73ba_isrForCrosstalkSources_996710_38.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash b942394cb85eb17984fe0555ff272c96 parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/c7fc680d-0249-4e46-841a-eb5dde2613f8_isrForCrosstalkSources_996718_25.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/c7fc680d-0249-4e46-841a-eb5dde2613f8_isrForCrosstalkSources_996718_25.stderr parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash b2a48728672e21dd2c51d8999ecd155c parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9ac3aeb6-d2da-463d-8050-1b9186fa844f_isrForCrosstalkSources_996706_17.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9ac3aeb6-d2da-463d-8050-1b9186fa844f_isrForCrosstalkSources_996706_17.stderr parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash b71a78ff6ebb547636f9f2b672a74dc5 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9072f4db-f1f4-4dac-afb2-410c4362b8d8_isrForCrosstalkSources_996718_62.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9072f4db-f1f4-4dac-afb2-410c4362b8d8_isrForCrosstalkSources_996718_62.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 8b6815e96ea15ee0e17ca6c8e06c44e2 parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/8b0dff56-199f-4191-b8ae-3757fff179b3_isrForCrosstalkSources_996714_8.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/8b0dff56-199f-4191-b8ae-3757fff179b3_isrForCrosstalkSources_996714_8.stderr parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 6f361920006c4c887bdd00b258e9a4f0 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/af10fd39-c53a-4ac7-8174-9cc939d89ffe_isrForCrosstalkSources_996709_59.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/af10fd39-c53a-4ac7-8174-9cc939d89ffe_isrForCrosstalkSources_996709_59.stderr parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 3c05e324fa48972e5a8f44a229b84306 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/d65d8366-293f-4c20-a4cb-25624f913d2b_isrForCrosstalkSources_996712_42.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/d65d8366-293f-4c20-a4cb-25624f913d2b_isrForCrosstalkSources_996712_42.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 9bf657e2d8426e9c434487668a85180e parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/11cfe9d1-02e4-4afc-8b51-44bdd720ce7d_isrForCrosstalkSources_996705_13.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/11cfe9d1-02e4-4afc-8b51-44bdd720ce7d_isrForCrosstalkSources_996705_13.stderr parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 32adc84847064dc0c9e1899e6b28d6e4 parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/c443aadc-9293-4138-8c0e-1b86c8977c70_isrForCrosstalkSources_996716_50.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/c443aadc-9293-4138-8c0e-1b86c8977c70_isrForCrosstalkSources_996716_50.stderr parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 5ee765c1d93b7712bd7c166156f65fbd parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/3cbf7ff4-ccc0-4b07-8422-b3dd7727de21_isrForCrosstalkSources_996716_33.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/3cbf7ff4-ccc0-4b07-8422-b3dd7727de21_isrForCrosstalkSources_996716_33.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash 28a50661907adc4c8f65b709fd474905 parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/9d776dc8-a68a-4064-ae7f-ee253de30b22_isrForCrosstalkSources_996705_25.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/9d776dc8-a68a-4064-ae7f-ee253de30b22_isrForCrosstalkSources_996705_25.stderr parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 5fd9ae7737396e5c33ed6ddd627bacee parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/50d6fe96-5f5c-4855-aa4f-74daf7a4abae_isrForCrosstalkSources_996715_21.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/50d6fe96-5f5c-4855-aa4f-74daf7a4abae_isrForCrosstalkSources_996715_21.stderr parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash debd352a6f7f31720b0e02486b22da1c parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/773f59b6-2997-4f7f-a8e4-6f960a92325a_isrForCrosstalkSources_996721_4.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/773f59b6-2997-4f7f-a8e4-6f960a92325a_isrForCrosstalkSources_996721_4.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 17e41db94c7bf6e710d7aec56e8c57bd parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/553bc1e9-a317-4486-9530-cd4167cd0a2c_isrForCrosstalkSources_996718_12.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/553bc1e9-a317-4486-9530-cd4167cd0a2c_isrForCrosstalkSources_996718_12.stderr parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash 47579f8500c3316e2cc8851a1fc91d9c parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/001bee0a-a7b3-48d9-85c2-e764a3ba5fad_isrForCrosstalkSources_996715_56.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/001bee0a-a7b3-48d9-85c2-e764a3ba5fad_isrForCrosstalkSources_996715_56.stderr parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash e3e03d001b9b4d4da85faa29da2861ae parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/794a919a-d970-4f9e-8f68-1a9b36080168_isrForCrosstalkSources_996709_20.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/794a919a-d970-4f9e-8f68-1a9b36080168_isrForCrosstalkSources_996709_20.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 386fdc7ccec917f1fac0539b8a932cd4 parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3bffb26e-395c-4c9c-b1bc-dad3dd7b6985_isrForCrosstalkSources_996706_40.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3bffb26e-395c-4c9c-b1bc-dad3dd7b6985_isrForCrosstalkSources_996706_40.stderr parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 22ac8eb976c38c04cd4ef372bfd9f45f parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b1753b7c-8bf9-44cd-9599-e249a37b0fe2_isrForCrosstalkSources_996713_14.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b1753b7c-8bf9-44cd-9599-e249a37b0fe2_isrForCrosstalkSources_996713_14.stderr parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash f6e975f5ad9251ea5861c37ab4f0570f parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/299e6579-ee3e-42b6-bc24-113e7ad490bd_isrForCrosstalkSources_996707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/299e6579-ee3e-42b6-bc24-113e7ad490bd_isrForCrosstalkSources_996707_7.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 0e15f9c61ee6cd1a5a10f1112e6c6fb3 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/993d4deb-d803-4aef-96ae-624f06d14eda_isrForCrosstalkSources_996718_40.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/993d4deb-d803-4aef-96ae-624f06d14eda_isrForCrosstalkSources_996718_40.stderr parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash c331dc094f06802aa46f45bd6ebb3fca parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76df380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/98c81e80-bfb6-46f6-bcd1-7691becf86a5_isrForCrosstalkSources_996719_8.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/98c81e80-bfb6-46f6-bcd1-7691becf86a5_isrForCrosstalkSources_996719_8.stderr parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash 934c13ed762cd3c46c1264872950ed0e parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7676020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/b44247d4-e615-4a07-8137-e01690d5161a_isrForCrosstalkSources_996720_17.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/b44247d4-e615-4a07-8137-e01690d5161a_isrForCrosstalkSources_996720_17.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash a6a1b87e3be38763523cb423a6f0ed33 parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/54b2f312-7960-494f-9ea0-ab6e4eece3f2_isrForCrosstalkSources_996706_58.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/54b2f312-7960-494f-9ea0-ab6e4eece3f2_isrForCrosstalkSources_996706_58.stderr parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 7cd6e6bf3a3e489b1723e792363a2f21 parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76771a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/476ce46b-2baa-4a9c-8a6f-4b1c7605398f_isrForCrosstalkSources_996710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/476ce46b-2baa-4a9c-8a6f-4b1c7605398f_isrForCrosstalkSources_996710_15.stderr parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash ebd1c6f02cda08ee8ad1761e51bbad3d parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/b2925e7e-8aad-419f-acc2-d87d360dc15c_isrForCrosstalkSources_996714_58.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/b2925e7e-8aad-419f-acc2-d87d360dc15c_isrForCrosstalkSources_996714_58.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash f734eee24fdf5aa5edac2bade27b3755 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7677100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cab1efc6-f687-414f-9cda-9cbdd7526bd7_isrForCrosstalkSources_996710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cab1efc6-f687-414f-9cda-9cbdd7526bd7_isrForCrosstalkSources_996710_14.stderr parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash 64540f7429b0b53cc48fc0b33541b116 parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/2c404240-1658-4806-9100-645388576e4e_isrForCrosstalkSources_996721_2.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/2c404240-1658-4806-9100-645388576e4e_isrForCrosstalkSources_996721_2.stderr parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 9b140101337b7607526b785de60cd8c5 parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/aad43998-9116-4a61-8460-c699431bcb57_isrForCrosstalkSources_996714_52.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/aad43998-9116-4a61-8460-c699431bcb57_isrForCrosstalkSources_996714_52.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash ef6a215a3bd85774ede3d393b9b2f014 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/9c670123-9eee-465c-9ee4-74f61f033383_isrForCrosstalkSources_996715_45.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/9c670123-9eee-465c-9ee4-74f61f033383_isrForCrosstalkSources_996715_45.stderr parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash 77cab8224d09ee913f34ba8e8ff37920 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f1244c99-b13c-418c-b452-0f2b50f0b1f5_isrForCrosstalkSources_996712_19.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f1244c99-b13c-418c-b452-0f2b50f0b1f5_isrForCrosstalkSources_996712_19.stderr parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 3da1d0efec3785e0b9e2dfcbd2c86e4d parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/2b74e048-3337-4f5a-8fbe-bab6d1073bc2_isrForCrosstalkSources_996705_22.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/2b74e048-3337-4f5a-8fbe-bab6d1073bc2_isrForCrosstalkSources_996705_22.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash 803bfed5a57ad6d2473ea34aabb73557 parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/078459c7-6600-479e-ad0b-1dc02bbb11aa_isrForCrosstalkSources_996721_12.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/078459c7-6600-479e-ad0b-1dc02bbb11aa_isrForCrosstalkSources_996721_12.stderr parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 0820280fbfcb9c8e9e8ab092235091ed parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5fdd79b8-aec1-4eb8-8b01-a87aa55a18c8_isrForCrosstalkSources_996720_55.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5fdd79b8-aec1-4eb8-8b01-a87aa55a18c8_isrForCrosstalkSources_996720_55.stderr parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 34edc04bcafdc9249c28d96911d68db4 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4b2317fa-9bd5-43c5-a22d-e8458af40867_isrForCrosstalkSources_996713_56.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4b2317fa-9bd5-43c5-a22d-e8458af40867_isrForCrosstalkSources_996713_56.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash d71e69dbd21b8656d6b3e80638aeceb8 parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3faaa108-90a5-47fb-9f2a-ea6a8e35eebc_isrForCrosstalkSources_996720_46.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3faaa108-90a5-47fb-9f2a-ea6a8e35eebc_isrForCrosstalkSources_996720_46.stderr parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash f3d0865f84744c33ec663ce9d059dff0 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9ffccccf-c379-45b6-b50f-46ff71b9d0cb_isrForCrosstalkSources_996714_42.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9ffccccf-c379-45b6-b50f-46ff71b9d0cb_isrForCrosstalkSources_996714_42.stderr parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 4fdb663313f34b79854802bbd6e45260 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/7e93dbda-5db7-4da1-8c9b-011d533bc97e_isrForCrosstalkSources_996720_24.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/7e93dbda-5db7-4da1-8c9b-011d533bc97e_isrForCrosstalkSources_996720_24.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash a279f4d1062c7b000baf31292cb97555 parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/14117787-ca01-428f-980e-d68a4c16c22a_isrForCrosstalkSources_996710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/14117787-ca01-428f-980e-d68a4c16c22a_isrForCrosstalkSources_996710_60.stderr parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash cdf681b74d5387f09008ad7a0d238169 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f07c29ba-061d-4082-aa19-619807450a5a_isrForCrosstalkSources_996709_31.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f07c29ba-061d-4082-aa19-619807450a5a_isrForCrosstalkSources_996709_31.stderr parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash ae963ebd54b16300170feb205e6b42fd parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/15471220-df71-436b-8439-b3048faaf34a_isrForCrosstalkSources_996710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/15471220-df71-436b-8439-b3048faaf34a_isrForCrosstalkSources_996710_9.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 6a627661f3fc2fb3eb04f7549f5360ca parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:55 proc_lsst.multi:170 [INFO] [multi] job_id 17071136 proc_lsst.multi INFO: [multi] job_id 17071136 2024-03-15 12:16:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 12:16:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071136 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071136 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17071136 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 881 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-15 12:16:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/a42756f2-ac01-4383-933f-7507aeab5a69_isrForCrosstalkSources_996717_47.stdout proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-15 12:16:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/a42756f2-ac01-4383-933f-7507aeab5a69_isrForCrosstalkSources_996717_47.stderr proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 parsl.dataflow.dflow DEBUG: Adding output dependencies proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 881 submitted for App isrForCrosstalkSources, not waiting on any dependency proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 12:16:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 2f755c6caca869595d21094d7ee8a461 parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76772e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/025a36e4-faac-4c7c-9210-b8bd1dbfaac9_isrForCrosstalkSources_996716_4.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/025a36e4-faac-4c7c-9210-b8bd1dbfaac9_isrForCrosstalkSources_996716_4.stderr parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 4b064b6c05513e1e0d08e09d94af6718 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c76dc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dba5533c-5b3e-45cd-9dd4-3ef4e08f9009_isrForCrosstalkSources_996711_30.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dba5533c-5b3e-45cd-9dd4-3ef4e08f9009_isrForCrosstalkSources_996711_30.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 6eb165b8285265cbe5175c1ed4be596d parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e68921bf-8395-4b91-8cc1-fd50a3f0d1eb_isrForCrosstalkSources_996720_1.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e68921bf-8395-4b91-8cc1-fd50a3f0d1eb_isrForCrosstalkSources_996720_1.stderr parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 075fdd8600bc6b37719029d3be563e41 parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/82185aae-c485-4096-9d61-90520f77b1dd_isrForCrosstalkSources_996711_10.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/82185aae-c485-4096-9d61-90520f77b1dd_isrForCrosstalkSources_996711_10.stderr parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash e652b3171cd920c16bbada53aa746e17 parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/0f233dea-4342-421d-985f-dd0d539e55c6_isrForCrosstalkSources_996708_10.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/0f233dea-4342-421d-985f-dd0d539e55c6_isrForCrosstalkSources_996708_10.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 69837b045f989670093c7af2a31c922a parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/6a865d33-4cb0-4725-bc8d-dda0bcc02d71_isrForCrosstalkSources_996705_7.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/6a865d33-4cb0-4725-bc8d-dda0bcc02d71_isrForCrosstalkSources_996705_7.stderr parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 339969e1afaf70f675cf17770ff696c1 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c61e760e-3ac6-493b-84ca-8e0b871a5cae_isrForCrosstalkSources_996713_20.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c61e760e-3ac6-493b-84ca-8e0b871a5cae_isrForCrosstalkSources_996713_20.stderr parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash adbe2319648468222925d7a09516f1bd parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/0aa61b9b-f766-4830-bff7-1bc4ad368d35_isrForCrosstalkSources_996709_25.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/0aa61b9b-f766-4830-bff7-1bc4ad368d35_isrForCrosstalkSources_996709_25.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 8652cba6bb0e0de459c733f23d598aed parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/2e560f20-9960-4b50-abbd-6abcdfdd342d_isrForCrosstalkSources_996707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/2e560f20-9960-4b50-abbd-6abcdfdd342d_isrForCrosstalkSources_996707_37.stderr parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 75e06b2154893e5a66552b8a23b3d64e parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/addf3674-b1f6-46ce-adf9-0acb9570081f_isrForCrosstalkSources_996708_3.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/addf3674-b1f6-46ce-adf9-0acb9570081f_isrForCrosstalkSources_996708_3.stderr parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash c0954239ccffb92c406db20d57d25084 parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/56576489-5e98-4f60-9966-f2d521a90ba6_isrForCrosstalkSources_996712_62.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/56576489-5e98-4f60-9966-f2d521a90ba6_isrForCrosstalkSources_996712_62.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash ed8fc6c7491ec491d6c005039ee7e8d3 parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cc20d248-faca-40ca-ac5a-505c26293035_isrForCrosstalkSources_996710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cc20d248-faca-40ca-ac5a-505c26293035_isrForCrosstalkSources_996710_40.stderr parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 3c0a4bb74cdd4f75a0e1718d7b434987 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/68727362-09b9-4f1a-b09a-15110e22edd3_isrForCrosstalkSources_996717_19.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/68727362-09b9-4f1a-b09a-15110e22edd3_isrForCrosstalkSources_996717_19.stderr parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash 756d151fd046d14003cdaa0cd5ed4c66 parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/48ca20fa-dc5e-4fa1-abee-357de42eb19e_isrForCrosstalkSources_996708_49.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/48ca20fa-dc5e-4fa1-abee-357de42eb19e_isrForCrosstalkSources_996708_49.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash e54661ebaac4129d6e8f6ce3c95007b6 parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/0ebe804f-b927-4d8d-8629-651a4fd1a256_isrForCrosstalkSources_996711_20.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/0ebe804f-b927-4d8d-8629-651a4fd1a256_isrForCrosstalkSources_996711_20.stderr parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash 9ecfd3c949ac3fd01861ffce4a445045 parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b1f947d5-108b-4d4e-8524-3d7dc5093426_isrForCrosstalkSources_996712_41.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b1f947d5-108b-4d4e-8524-3d7dc5093426_isrForCrosstalkSources_996712_41.stderr parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash d4598b4acaeb491ec4a9ae0be4303f26 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/9ec7f57c-5f4a-4d60-9795-69f9d6c1e39b_isrForCrosstalkSources_996721_22.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/9ec7f57c-5f4a-4d60-9795-69f9d6c1e39b_isrForCrosstalkSources_996721_22.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 9b9be8b5df68f109a6c5c54ac175cd5c parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/1a567327-a6e0-44c9-8d3c-192f1cf13075_isrForCrosstalkSources_996714_12.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/1a567327-a6e0-44c9-8d3c-192f1cf13075_isrForCrosstalkSources_996714_12.stderr parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 9239e06376f8c7d38574d36bc257a953 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/bcb18af3-b0e8-469f-8198-3d67a2ea299e_isrForCrosstalkSources_996709_9.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/bcb18af3-b0e8-469f-8198-3d67a2ea299e_isrForCrosstalkSources_996709_9.stderr parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 4966b8b1b3fdd5fe9b43ae223f7ff9fc parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c6be1a3d-5259-4b7a-ba1c-a543e3c4e2c6_isrForCrosstalkSources_996705_51.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c6be1a3d-5259-4b7a-ba1c-a543e3c4e2c6_isrForCrosstalkSources_996705_51.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 648b7f2e4410e5f7d664222dd9ea12fc parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/aa56a0bb-2c11-4d84-960e-12fdd5618a83_isrForCrosstalkSources_996706_23.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/aa56a0bb-2c11-4d84-960e-12fdd5618a83_isrForCrosstalkSources_996706_23.stderr parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash d25996650aff038de043d09b9d6c46ed parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/7a5ae8b1-f41f-4e98-a136-22812d10a9f1_isrForCrosstalkSources_996710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/7a5ae8b1-f41f-4e98-a136-22812d10a9f1_isrForCrosstalkSources_996710_62.stderr parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 7ce699a57bdf63792a73c000f6a7c5b7 parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d404d5d3-6215-450f-892c-3ed3e0d98e24_isrForCrosstalkSources_996708_19.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d404d5d3-6215-450f-892c-3ed3e0d98e24_isrForCrosstalkSources_996708_19.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash 197d041cf3929b4b043679b56d4547f4 parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d0c8cca8-7483-48f9-9d33-b26a7414f6d5_isrForCrosstalkSources_996708_2.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d0c8cca8-7483-48f9-9d33-b26a7414f6d5_isrForCrosstalkSources_996708_2.stderr parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 09342d1fc4b20d4b004c669abc773ebb parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f6c99c6b-f533-4aee-be1e-33505aa17724_isrForCrosstalkSources_996708_38.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f6c99c6b-f533-4aee-be1e-33505aa17724_isrForCrosstalkSources_996708_38.stderr parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash acf36bb0b746fdeb6ecc337cf02b68c0 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cee31caf-a3ef-401d-950f-e0dfde9eb5b8_isrForCrosstalkSources_996714_55.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cee31caf-a3ef-401d-950f-e0dfde9eb5b8_isrForCrosstalkSources_996714_55.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 31914ca1c93cea56daee9bda5118971d parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/ed12b334-f885-49ea-8a2e-06bc2923df4a_isrForCrosstalkSources_996712_38.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/ed12b334-f885-49ea-8a2e-06bc2923df4a_isrForCrosstalkSources_996712_38.stderr parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash 1df06f8a4ac93760d316851f3b6231e5 parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/bac5dfe4-764a-4ecc-816f-bf641b8d4849_isrForCrosstalkSources_996707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/bac5dfe4-764a-4ecc-816f-bf641b8d4849_isrForCrosstalkSources_996707_57.stderr parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash c3dcce300a7af4344267975770a2449b parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e50b19f3-1337-4d00-8d66-6d01af21269e_isrForCrosstalkSources_996709_43.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e50b19f3-1337-4d00-8d66-6d01af21269e_isrForCrosstalkSources_996709_43.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash ca596e7014a543426b4416db78b8975c parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/64c2dc23-6eef-481d-8590-ad60cd1f45d8_isrForCrosstalkSources_996720_35.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/64c2dc23-6eef-481d-8590-ad60cd1f45d8_isrForCrosstalkSources_996720_35.stderr parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 950daddc7929bd81e7c9a31377345a84 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/818eb3c1-546b-49e5-bf0b-d545ac1dc791_isrForCrosstalkSources_996707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/818eb3c1-546b-49e5-bf0b-d545ac1dc791_isrForCrosstalkSources_996707_38.stderr parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 743cff5eadabc3f0457664b1af901979 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/3276a900-49a2-495c-bdaf-dd72d1a263f1_isrForCrosstalkSources_996710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/3276a900-49a2-495c-bdaf-dd72d1a263f1_isrForCrosstalkSources_996710_16.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 5ea5674efbee4d31038fe4f329c46002 parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/116e44f0-80b7-4c0d-aa87-c672624cee73_isrForCrosstalkSources_996706_25.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/116e44f0-80b7-4c0d-aa87-c672624cee73_isrForCrosstalkSources_996706_25.stderr parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash f263dfbe82ebd6611908a873d9f8933e parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4df4f8c8-df2f-46ce-aebf-5e9a56ee985d_isrForCrosstalkSources_996714_1.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4df4f8c8-df2f-46ce-aebf-5e9a56ee985d_isrForCrosstalkSources_996714_1.stderr parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash f5a8e4736e9ac24d317293f8a9382a17 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e885c27c-1659-4b11-b6da-84f17ba5f61e_isrForCrosstalkSources_996721_31.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e885c27c-1659-4b11-b6da-84f17ba5f61e_isrForCrosstalkSources_996721_31.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash 407161939cad377165a582e132fdc927 parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a2287f53-508e-488d-9bdb-b5bedd897255_isrForCrosstalkSources_996719_57.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a2287f53-508e-488d-9bdb-b5bedd897255_isrForCrosstalkSources_996719_57.stderr parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 6b76e94e8d95126bc61694d5b3cfe5ed parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/52b8a47a-7182-4269-b49d-8245a056738b_isrForCrosstalkSources_996719_5.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/52b8a47a-7182-4269-b49d-8245a056738b_isrForCrosstalkSources_996719_5.stderr parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 115c4a6a1984862e9216d17f5019e320 parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/42d48cab-5627-4dec-962f-32f87b8d377d_isrForCrosstalkSources_996718_32.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/42d48cab-5627-4dec-962f-32f87b8d377d_isrForCrosstalkSources_996718_32.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash b938ed6c71311fdd75d969c43d70a9b3 parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/558161fc-853f-44fe-abcc-0979a3b146eb_isrForCrosstalkSources_996713_62.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/558161fc-853f-44fe-abcc-0979a3b146eb_isrForCrosstalkSources_996713_62.stderr parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash 9f845ea772526b863f130db9a261fed9 parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1aa5e52e-fcaa-4dce-9606-ef0e9800787d_isrForCrosstalkSources_996707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1aa5e52e-fcaa-4dce-9606-ef0e9800787d_isrForCrosstalkSources_996707_24.stderr parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 220e7e8bd85ba93f420b8313727ab284 parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/87f3aa4e-5686-48da-a7fb-e16342600805_isrForCrosstalkSources_996709_48.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/87f3aa4e-5686-48da-a7fb-e16342600805_isrForCrosstalkSources_996709_48.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash a83ddfc2f2dffd43a20e29a0d97c7cac parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f46c1d50-a0fd-4601-ba56-2117db0086b3_isrForCrosstalkSources_996714_28.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f46c1d50-a0fd-4601-ba56-2117db0086b3_isrForCrosstalkSources_996714_28.stderr parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash f5ae6039a898da3fb9a815b168f018fb parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/829e83bc-fa9e-49e6-bf3d-1da1f3b3bfa8_isrForCrosstalkSources_996721_13.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/829e83bc-fa9e-49e6-bf3d-1da1f3b3bfa8_isrForCrosstalkSources_996721_13.stderr parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash b6e4d0cc5ccfb00ca6ae1c36d062084a parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3fa0c897-5f4b-43ce-a551-1176a17ec879_isrForCrosstalkSources_996720_33.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3fa0c897-5f4b-43ce-a551-1176a17ec879_isrForCrosstalkSources_996720_33.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash f3fe741bbc2425eabec5b354b4d56d92 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/c5574831-d42a-425c-a909-d928ad1bcbea_isrForCrosstalkSources_996716_51.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/c5574831-d42a-425c-a909-d928ad1bcbea_isrForCrosstalkSources_996716_51.stderr parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 63f3c3ab9b3431623ed8c429442804a4 parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/a4d6c9a6-4d62-47cb-9f79-d1c4e6b2987f_isrForCrosstalkSources_996717_4.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/a4d6c9a6-4d62-47cb-9f79-d1c4e6b2987f_isrForCrosstalkSources_996717_4.stderr parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 9083c271a32ae33645d867310407727b parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3c232d6d-1025-4bb6-9b45-bcc5a379c948_isrForCrosstalkSources_996705_43.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3c232d6d-1025-4bb6-9b45-bcc5a379c948_isrForCrosstalkSources_996705_43.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 971828a69af23be3933e3354b66f3125 parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a4b23e60-7614-428e-b11a-6b0275944a6e_isrForCrosstalkSources_996708_13.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a4b23e60-7614-428e-b11a-6b0275944a6e_isrForCrosstalkSources_996708_13.stderr parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash 7b729d1eb52387c7b4b1daf946b5e5ae parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5a3cb6e2-ddfd-46fa-9683-312c31777722_isrForCrosstalkSources_996709_16.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5a3cb6e2-ddfd-46fa-9683-312c31777722_isrForCrosstalkSources_996709_16.stderr parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash feec2a8f2d3e6a343f641d51c1d2116f parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1b3145b0-40ea-4aa6-8520-bb0b28c879d4_isrForCrosstalkSources_996721_16.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1b3145b0-40ea-4aa6-8520-bb0b28c879d4_isrForCrosstalkSources_996721_16.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash 6d85a54dc00d9f07258c94368883b740 parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0954d1e3-d773-43b4-8c47-0b377c3b8171_isrForCrosstalkSources_996719_19.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0954d1e3-d773-43b4-8c47-0b377c3b8171_isrForCrosstalkSources_996719_19.stderr parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash c570c09421ef7d871b586316f3bc704e parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/198b5a5a-9023-492b-adfa-79bd4733ddb6_isrForCrosstalkSources_996708_42.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/198b5a5a-9023-492b-adfa-79bd4733ddb6_isrForCrosstalkSources_996708_42.stderr parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash 6905e31d778f34f95effb64772192da3 parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8862ed49-21e5-4424-a5e4-bb8851511dd5_isrForCrosstalkSources_996717_8.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8862ed49-21e5-4424-a5e4-bb8851511dd5_isrForCrosstalkSources_996717_8.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 7e06fe90573fdffeb5dd931d3cfbcc27 parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/116066c0-0067-4023-b57e-409bdedd2dfb_isrForCrosstalkSources_996714_43.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/116066c0-0067-4023-b57e-409bdedd2dfb_isrForCrosstalkSources_996714_43.stderr parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash d47e122d00c28b6927ed58c95dd5df00 parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/69deb6a3-2ad0-41b9-bfc0-31075ec1d936_isrForCrosstalkSources_996718_44.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/69deb6a3-2ad0-41b9-bfc0-31075ec1d936_isrForCrosstalkSources_996718_44.stderr parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash 27459eb4580ebc691f4442c3c6b8fdd2 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8df5573b-fb37-4fcf-9958-239e0624387e_isrForCrosstalkSources_996721_20.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8df5573b-fb37-4fcf-9958-239e0624387e_isrForCrosstalkSources_996721_20.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash f4312ae5ef3bc9bc3a719c3bbfe5cb41 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e962f6ca-e683-4f8d-909b-e8190ad63a78_isrForCrosstalkSources_996713_48.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e962f6ca-e683-4f8d-909b-e8190ad63a78_isrForCrosstalkSources_996713_48.stderr parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 761f4df54042825dc0253037eb20b222 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c74f6e5f-50ae-4dcd-829f-7122751c3085_isrForCrosstalkSources_996706_44.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c74f6e5f-50ae-4dcd-829f-7122751c3085_isrForCrosstalkSources_996706_44.stderr parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 1bf3f2edc2d11e0e89146cd46fc0bf8a parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/6bdc18a1-76f5-4965-add4-2b3a6ff3bbdb_isrForCrosstalkSources_996720_50.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/6bdc18a1-76f5-4965-add4-2b3a6ff3bbdb_isrForCrosstalkSources_996720_50.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 768f694c5ee8c02fade8f379ae6ebb21 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/dca8013d-b107-4c1f-a1c6-5d24dbb3b5f0_isrForCrosstalkSources_996705_59.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/dca8013d-b107-4c1f-a1c6-5d24dbb3b5f0_isrForCrosstalkSources_996705_59.stderr parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 615ea1b2d67dbbf19d7c07287515bff4 parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/060ac05c-18bd-4f5e-9a47-aa6a3c6dec2f_isrForCrosstalkSources_996705_54.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/060ac05c-18bd-4f5e-9a47-aa6a3c6dec2f_isrForCrosstalkSources_996705_54.stderr parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 757bbd1c9520262fba2a4d2416b229c4 parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/756da7d0-2429-423c-9cc4-e7c1abc37d22_isrForCrosstalkSources_996719_61.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/756da7d0-2429-423c-9cc4-e7c1abc37d22_isrForCrosstalkSources_996719_61.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 41ec6b25d5c086f4272ba581955ffd28 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4a3dc0fe-b902-4d85-9c17-be4bd72c0ae8_isrForCrosstalkSources_996713_10.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4a3dc0fe-b902-4d85-9c17-be4bd72c0ae8_isrForCrosstalkSources_996713_10.stderr parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 86e3fef0aa04c464ee5b49cbe1928263 parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/78148eec-7ec6-47d5-8f1f-37a5adcb305f_isrForCrosstalkSources_996714_59.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/78148eec-7ec6-47d5-8f1f-37a5adcb305f_isrForCrosstalkSources_996714_59.stderr parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 901172b6efcee080a48872366e58533f parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/81d5cd61-9e8f-4cf4-a754-98415608a768_isrForCrosstalkSources_996721_47.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/81d5cd61-9e8f-4cf4-a754-98415608a768_isrForCrosstalkSources_996721_47.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash c35410d0e6a2a4bdfa641d26f4a06310 parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5b2d3e85-f012-422a-a422-a4fea49c7dc3_isrForCrosstalkSources_996721_46.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5b2d3e85-f012-422a-a422-a4fea49c7dc3_isrForCrosstalkSources_996721_46.stderr parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash a9af8f9ead8f16bd094872775a87da74 parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2f30d21e-626b-46fd-99ee-5272dd14e736_isrForCrosstalkSources_996714_30.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2f30d21e-626b-46fd-99ee-5272dd14e736_isrForCrosstalkSources_996714_30.stderr parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 09ae6e37d88672c27edd0114968f65af parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5dff0b34-25f0-4bf4-bcad-50c70eb91770_isrForCrosstalkSources_996719_52.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5dff0b34-25f0-4bf4-bcad-50c70eb91770_isrForCrosstalkSources_996719_52.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 3b7f0d0f705d21d71c43037fcd90416f parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/3291973a-2117-4962-8b80-769cb9398c0f_isrForCrosstalkSources_996718_42.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/3291973a-2117-4962-8b80-769cb9398c0f_isrForCrosstalkSources_996718_42.stderr parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash 98e708049def22517fc70d2c3c201f25 parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/796f2390-d6c6-4878-99df-4ed13a61f34f_isrForCrosstalkSources_996712_59.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/796f2390-d6c6-4878-99df-4ed13a61f34f_isrForCrosstalkSources_996712_59.stderr parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash c8cf6ed8b5778116f26719fa953774f2 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/aa04b811-4aa6-4258-9657-339b30a54440_isrForCrosstalkSources_996719_55.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/aa04b811-4aa6-4258-9657-339b30a54440_isrForCrosstalkSources_996719_55.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 49e2e85c488b224d8937206f2ca1f361 parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/53f23b29-5ea2-45e6-a849-aab3cc4897eb_isrForCrosstalkSources_996716_38.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/53f23b29-5ea2-45e6-a849-aab3cc4897eb_isrForCrosstalkSources_996716_38.stderr parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash fcb070133417e6e4b06a78f727c33456 parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/0b54b89d-17a2-4064-9500-2a572833c413_isrForCrosstalkSources_996717_10.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/0b54b89d-17a2-4064-9500-2a572833c413_isrForCrosstalkSources_996717_10.stderr parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 26bc863c5b881197ad66bc9b9c9042f9 parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/932c3d52-b883-46f3-9eff-1dbd2e22077f_isrForCrosstalkSources_996710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/932c3d52-b883-46f3-9eff-1dbd2e22077f_isrForCrosstalkSources_996710_42.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 0857cba031a7997dd4ebbef1f1091fe9 parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/4dd49ead-241a-487f-b8d3-eb5d9259ab30_isrForCrosstalkSources_996708_62.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/4dd49ead-241a-487f-b8d3-eb5d9259ab30_isrForCrosstalkSources_996708_62.stderr parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 7259b17622d43e944601ab63b0912922 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8ac1c605-635f-4630-9814-e47161dcdd9a_isrForCrosstalkSources_996717_9.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8ac1c605-635f-4630-9814-e47161dcdd9a_isrForCrosstalkSources_996717_9.stderr parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 0a3868dd884bd6f0ee6d3fdb9631607b parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/83e88f25-13e8-4e9b-aa35-219ed4109a43_isrForCrosstalkSources_996712_37.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/83e88f25-13e8-4e9b-aa35-219ed4109a43_isrForCrosstalkSources_996712_37.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 7bd121250b2b0c331518d7147101d54c parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6a0ccf01-eb11-4439-a7a6-2ca76132355d_isrForCrosstalkSources_996711_7.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6a0ccf01-eb11-4439-a7a6-2ca76132355d_isrForCrosstalkSources_996711_7.stderr parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 16b5b4ab48b4a547d208b2808c8d8e6f parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f70a1644-19bc-4228-a4a8-f74a9a1cddc3_isrForCrosstalkSources_996713_26.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f70a1644-19bc-4228-a4a8-f74a9a1cddc3_isrForCrosstalkSources_996713_26.stderr parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash b712707a5208eeff7663dc3cd6a9db8a parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/98a3f2a7-1768-4823-b7e6-754752c29603_isrForCrosstalkSources_996716_28.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/98a3f2a7-1768-4823-b7e6-754752c29603_isrForCrosstalkSources_996716_28.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 286140843b05e94cb75924656c48d1ef parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5190a963-5d78-405c-b86c-c1e144dd8c51_isrForCrosstalkSources_996710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5190a963-5d78-405c-b86c-c1e144dd8c51_isrForCrosstalkSources_996710_4.stderr parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash d358b749a8ae00d92bd823050e49dffb parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/05ae7d62-fa36-4b31-9fee-d0d7b6a035be_isrForCrosstalkSources_996720_31.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/05ae7d62-fa36-4b31-9fee-d0d7b6a035be_isrForCrosstalkSources_996720_31.stderr parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash a09aec7138b352a3dc45dd035f41739a parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/2bf5aac3-9107-43d1-a72b-30652da090b3_isrForCrosstalkSources_996717_36.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/2bf5aac3-9107-43d1-a72b-30652da090b3_isrForCrosstalkSources_996717_36.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 9240015e3d00343307c3033a149a0dcc parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/93dfcd11-7899-49c8-a35a-c42e904668e4_isrForCrosstalkSources_996721_26.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/93dfcd11-7899-49c8-a35a-c42e904668e4_isrForCrosstalkSources_996721_26.stderr parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 3049a19b949ea072d6ca7bf478e468ef parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6a73b79d-123f-4860-ba77-ea631efa042f_isrForCrosstalkSources_996713_23.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6a73b79d-123f-4860-ba77-ea631efa042f_isrForCrosstalkSources_996713_23.stderr parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash 8832f3aa9755614e9a3878d38ddede2a parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/fd4f27ec-5d9f-4811-8268-42871fd79685_isrForCrosstalkSources_996720_59.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/fd4f27ec-5d9f-4811-8268-42871fd79685_isrForCrosstalkSources_996720_59.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 79ef7c06de754fd1c162ab2b27bf0463 parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74380e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ea01dff7-4fa6-4aa4-82a1-ace6b1db09f6_isrForCrosstalkSources_996716_56.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ea01dff7-4fa6-4aa4-82a1-ace6b1db09f6_isrForCrosstalkSources_996716_56.stderr parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash 4c5dfe071e013b2badf2b16debc4de3e parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/13c3bc86-6abb-4d87-aa6a-babe84ccf314_isrForCrosstalkSources_996709_27.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/13c3bc86-6abb-4d87-aa6a-babe84ccf314_isrForCrosstalkSources_996709_27.stderr parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash edb55dc2f9dbd6c9687451ad30957245 parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74382c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b94ba833-8ecd-42c1-9cea-827a2166711d_isrForCrosstalkSources_996706_60.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b94ba833-8ecd-42c1-9cea-827a2166711d_isrForCrosstalkSources_996706_60.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash f0daf8f4f428421f8f0eb96cce22fa7f parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:56 proc_lsst.multi:170 [INFO] [multi] job_id 17071137 proc_lsst.multi INFO: [multi] job_id 17071137 2024-03-15 12:16:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 12:16:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071137 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071137 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17071137 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-15 12:16:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-15 12:16:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 12:16:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b4208fca-f0e5-4499-bf2e-47380539325e_isrForCrosstalkSources_996712_33.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b4208fca-f0e5-4499-bf2e-47380539325e_isrForCrosstalkSources_996712_33.stderr parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 8609eabca05af811bf54b9119e0acf0a parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/dd6fdd3a-1907-4a27-9967-9ec8cbbcade7_isrForCrosstalkSources_996715_52.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/dd6fdd3a-1907-4a27-9967-9ec8cbbcade7_isrForCrosstalkSources_996715_52.stderr parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 24c05962ff1873fcfa80b43ee3af26e8 parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c75a0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/74302b86-8b83-41ee-9a2e-e76d12162fcb_isrForCrosstalkSources_996708_50.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/74302b86-8b83-41ee-9a2e-e76d12162fcb_isrForCrosstalkSources_996708_50.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 415d0500cf12afd1b60cb552f1978f6a parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ce6a269b-ebd7-49c1-a65a-f6ca58f3391f_isrForCrosstalkSources_996713_30.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ce6a269b-ebd7-49c1-a65a-f6ca58f3391f_isrForCrosstalkSources_996713_30.stderr parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 77aa843c5ba62add68b5f0633d6ecac1 parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/14dcbf69-3e97-479d-8e72-986613f26a78_isrForCrosstalkSources_996710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/14dcbf69-3e97-479d-8e72-986613f26a78_isrForCrosstalkSources_996710_48.stderr parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash 2b7ebcad8f536113701eb069707efdc6 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/dea16a6d-76f9-43d8-bf5c-0f8b7039606e_isrForCrosstalkSources_996710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/dea16a6d-76f9-43d8-bf5c-0f8b7039606e_isrForCrosstalkSources_996710_31.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 9259336ef8ded9aa9c6c76eac8ea94cb parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74385e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/475a3c76-71d1-4cfd-9dbf-33be9b28df6d_isrForCrosstalkSources_996709_58.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/475a3c76-71d1-4cfd-9dbf-33be9b28df6d_isrForCrosstalkSources_996709_58.stderr parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 3c86c62f724aa1210110913d2eb8a5f3 parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/888f7e7b-3317-4dbc-b449-ef7b36683164_isrForCrosstalkSources_996714_9.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/888f7e7b-3317-4dbc-b449-ef7b36683164_isrForCrosstalkSources_996714_9.stderr parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 1ec470fca66d39db289d97f4571d1b91 parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/99f0b31e-1aa3-4013-8ab4-2d522e50ceac_isrForCrosstalkSources_996717_3.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/99f0b31e-1aa3-4013-8ab4-2d522e50ceac_isrForCrosstalkSources_996717_3.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash a987d1abfed4538a2e8e356f66f1752c parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74387c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/04e77813-2255-4a4c-9b3c-c7d91a23ad09_isrForCrosstalkSources_996709_4.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/04e77813-2255-4a4c-9b3c-c7d91a23ad09_isrForCrosstalkSources_996709_4.stderr parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash d73bded55a9289f8327ce715ea26a5f6 parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/0a78e455-8d60-4a9b-a776-054fcc2e7a92_isrForCrosstalkSources_996718_60.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/0a78e455-8d60-4a9b-a776-054fcc2e7a92_isrForCrosstalkSources_996718_60.stderr parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 7c9fe310fa805acec27c5f0916def5f5 parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8523ba52-3e5d-4784-9dbf-7f86e1aa5b2e_isrForCrosstalkSources_996718_7.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8523ba52-3e5d-4784-9dbf-7f86e1aa5b2e_isrForCrosstalkSources_996718_7.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash d38b59f3729d3432a2d0e27a83018fd0 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e8557416-925f-4740-850d-0118699e7536_isrForCrosstalkSources_996716_16.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e8557416-925f-4740-850d-0118699e7536_isrForCrosstalkSources_996716_16.stderr parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash ca2fd2d199fcd7a33052a539f8683ab5 parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/079515be-1b1c-4fd5-91c7-469cf774f8c6_isrForCrosstalkSources_996716_22.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/079515be-1b1c-4fd5-91c7-469cf774f8c6_isrForCrosstalkSources_996716_22.stderr parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash 03cc4ee93b9165cfc19c967f4241c8a3 parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f3459cb8-029c-40c4-9e3a-15f149752114_isrForCrosstalkSources_996705_18.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f3459cb8-029c-40c4-9e3a-15f149752114_isrForCrosstalkSources_996705_18.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 5f13665699a0df09b62ed814ca986351 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74389a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/61b82d35-be8a-4af2-b10b-667cd46af7cc_isrForCrosstalkSources_996721_35.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/61b82d35-be8a-4af2-b10b-667cd46af7cc_isrForCrosstalkSources_996721_35.stderr parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 73ca0c8fc2453cb813c5c1114dc9be63 parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8683ead5-63a0-4757-95b3-24584fadec9e_isrForCrosstalkSources_996715_6.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8683ead5-63a0-4757-95b3-24584fadec9e_isrForCrosstalkSources_996715_6.stderr parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash c8a821f905ea8e989c8711b814e247f9 parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0566892c-25d4-4303-9078-4f2d15164f60_isrForCrosstalkSources_996720_39.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0566892c-25d4-4303-9078-4f2d15164f60_isrForCrosstalkSources_996720_39.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash ef277ec2f10e252758065631b7f3d13f parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b14dbc99-5d16-4e51-93b9-47d0e51bc4c5_isrForCrosstalkSources_996715_38.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b14dbc99-5d16-4e51-93b9-47d0e51bc4c5_isrForCrosstalkSources_996715_38.stderr parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 276edd151af144af41220c9102709a73 parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fd297f4c-4188-46a4-8eb5-cd2521bf421d_isrForCrosstalkSources_996717_7.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fd297f4c-4188-46a4-8eb5-cd2521bf421d_isrForCrosstalkSources_996717_7.stderr parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash 1214b207f3b66cbd17a47710fe341e7f parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/354c2e8d-5f15-4b92-a40c-676275d3fae1_isrForCrosstalkSources_996721_30.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/354c2e8d-5f15-4b92-a40c-676275d3fae1_isrForCrosstalkSources_996721_30.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 8bd6260b14b4f914a6d31da65f54b7ab parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/18b09049-0dab-4589-b168-4b0d93b3c331_isrForCrosstalkSources_996713_47.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/18b09049-0dab-4589-b168-4b0d93b3c331_isrForCrosstalkSources_996713_47.stderr parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 04fff77aaba0df38734f791fd5538958 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/40554e58-837e-42a4-a79e-33757dd2da4e_isrForCrosstalkSources_996717_27.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/40554e58-837e-42a4-a79e-33757dd2da4e_isrForCrosstalkSources_996717_27.stderr parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash ce6de91c106a7b82ad909b18036506ba parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f627e300-18f8-40e0-8c43-44ce57302045_isrForCrosstalkSources_996711_19.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f627e300-18f8-40e0-8c43-44ce57302045_isrForCrosstalkSources_996711_19.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 6d690acbfecc583adb4c29763f5ddf14 parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7438ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1d4f90a9-390a-47f7-ab23-340ffb20a7c5_isrForCrosstalkSources_996721_40.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1d4f90a9-390a-47f7-ab23-340ffb20a7c5_isrForCrosstalkSources_996721_40.stderr parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 0a8651cfcaef8ba642fc0a89e5d02d7a parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/bc25109c-c667-4f94-9ef6-2675536b2ab5_isrForCrosstalkSources_996706_45.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/bc25109c-c667-4f94-9ef6-2675536b2ab5_isrForCrosstalkSources_996706_45.stderr parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash 2727283d5f63549649b6078a73cf7f4a parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/68ceb28b-f2dc-4e8a-b778-7feb937bf31a_isrForCrosstalkSources_996721_9.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/68ceb28b-f2dc-4e8a-b778-7feb937bf31a_isrForCrosstalkSources_996721_9.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 0399c48f31fe6d69795fa16bd55e2722 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3d0be745-cdf4-42ea-b5a9-23dd02572825_isrForCrosstalkSources_996711_43.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3d0be745-cdf4-42ea-b5a9-23dd02572825_isrForCrosstalkSources_996711_43.stderr parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash dea961ec39534a43d1708e9f552e86da parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3b0df954-b1ff-4cc3-bc2e-b0c65c0bb961_isrForCrosstalkSources_996709_7.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3b0df954-b1ff-4cc3-bc2e-b0c65c0bb961_isrForCrosstalkSources_996709_7.stderr parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash c2c5cc4df2632827d19932df5c07cc6d parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74391c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/77a175ae-1e9a-4e15-845a-f46b7e729ec6_isrForCrosstalkSources_996707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/77a175ae-1e9a-4e15-845a-f46b7e729ec6_isrForCrosstalkSources_996707_52.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 64a8fd1eeab78677bd003a683f3d5716 parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74393a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/3e2aa1f8-a198-456b-b82d-4bd90e1e8424_isrForCrosstalkSources_996716_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/3e2aa1f8-a198-456b-b82d-4bd90e1e8424_isrForCrosstalkSources_996716_39.stderr parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 9f6d8f082d441b0043588460a94ecfd7 parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74394e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4af084bb-d510-45aa-8300-9ed069370230_isrForCrosstalkSources_996718_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4af084bb-d510-45aa-8300-9ed069370230_isrForCrosstalkSources_996718_56.stderr parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 82655bced0821ecf682a47a75cc9cac7 parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/5fe78bef-05f5-4f40-9826-35833dff657b_isrForCrosstalkSources_996713_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/5fe78bef-05f5-4f40-9826-35833dff657b_isrForCrosstalkSources_996713_55.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash b169eed7a8a742b90af720ab8c22f6d5 parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c9ce87ec-6f6c-40f3-9e5c-42af1daac7cc_isrForCrosstalkSources_996712_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c9ce87ec-6f6c-40f3-9e5c-42af1daac7cc_isrForCrosstalkSources_996712_28.stderr parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 28629fb72bc033779d149288a8aaa40e parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/1afa036c-8709-4407-824d-8e9649ad3f5b_isrForCrosstalkSources_996716_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/1afa036c-8709-4407-824d-8e9649ad3f5b_isrForCrosstalkSources_996716_3.stderr parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 22969e9bfbdc4fa17ddd9a3608fc0cdb parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/b6a9d2f7-f0a0-441b-9276-8bee65044694_isrForCrosstalkSources_996717_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/b6a9d2f7-f0a0-441b-9276-8bee65044694_isrForCrosstalkSources_996717_20.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash 27b503d43e64b8130f50a2958e757e2b parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9b4783e5-af54-46f2-aa55-19aac0585fe3_isrForCrosstalkSources_996714_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9b4783e5-af54-46f2-aa55-19aac0585fe3_isrForCrosstalkSources_996714_6.stderr parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 7682b430b6553b4fcd709ab819c2a4ea parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f52e5f0c-14a0-4f8c-9039-ea53ce9602fd_isrForCrosstalkSources_996705_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f52e5f0c-14a0-4f8c-9039-ea53ce9602fd_isrForCrosstalkSources_996705_24.stderr parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 38fde6c752fd7152ed3a53a1007e327f parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74398a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/36857785-af52-4228-9da9-1e4c56cfc64d_isrForCrosstalkSources_996707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/36857785-af52-4228-9da9-1e4c56cfc64d_isrForCrosstalkSources_996707_42.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 748a78fd2136868df80a2fb8b3a5c8a0 parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1fdfcedf-15be-4011-8322-93e3d439e70e_isrForCrosstalkSources_996712_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1fdfcedf-15be-4011-8322-93e3d439e70e_isrForCrosstalkSources_996712_61.stderr parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash 8d4423a70b72e78cf0095471405e5765 parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c74399e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/51868d7a-c92e-471a-b034-60c4167dd6b9_isrForCrosstalkSources_996708_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/51868d7a-c92e-471a-b034-60c4167dd6b9_isrForCrosstalkSources_996708_21.stderr parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash a979f272e9f1509fc3efd7559de4ac22 parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/df3aea11-6a47-428c-bb02-53e6e4ab5b07_isrForCrosstalkSources_996711_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/df3aea11-6a47-428c-bb02-53e6e4ab5b07_isrForCrosstalkSources_996711_62.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 75bc5eac9792b3f75038344e88396cbd parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5d8ba5fd-0c1d-4d5d-9ba4-8d12e7df34a4_isrForCrosstalkSources_996709_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5d8ba5fd-0c1d-4d5d-9ba4-8d12e7df34a4_isrForCrosstalkSources_996709_10.stderr parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 2efe9809a2f8a5b907906b912152fea3 parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/80bc3169-27d3-4072-8c4d-af7416c21bf4_isrForCrosstalkSources_996715_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/80bc3169-27d3-4072-8c4d-af7416c21bf4_isrForCrosstalkSources_996715_23.stderr parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash c4932d2bcc8bbfa2c70b29fb4919057e parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0c862462-4b11-4cd5-84ac-296eeba120fb_isrForCrosstalkSources_996720_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0c862462-4b11-4cd5-84ac-296eeba120fb_isrForCrosstalkSources_996720_61.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 26cec1e0eb276cd9d8d150e53db7c649 parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/aff884cc-ec89-4940-bc6f-6917358f657a_isrForCrosstalkSources_996717_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/aff884cc-ec89-4940-bc6f-6917358f657a_isrForCrosstalkSources_996717_21.stderr parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash cac61e5b9ab766e0ec86f1e33a3c6564 parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/70fda952-aede-461e-b6d9-6d028eb3f69f_isrForCrosstalkSources_996712_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/70fda952-aede-461e-b6d9-6d028eb3f69f_isrForCrosstalkSources_996712_58.stderr parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash e5d0d3a84cb530326f70fe0ad8caee10 parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/0f0cab85-3bb1-416f-b0ba-4ca251193c2f_isrForCrosstalkSources_996707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/0f0cab85-3bb1-416f-b0ba-4ca251193c2f_isrForCrosstalkSources_996707_5.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash 22c6dfe601fe7edab049df1ca69d6bdd parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c7f32692-bfb5-40f4-a611-1a7ea1743fdf_isrForCrosstalkSources_996709_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c7f32692-bfb5-40f4-a611-1a7ea1743fdf_isrForCrosstalkSources_996709_50.stderr parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 5fb539913036d11f8ee1ba1f797d717d parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/cd29273a-3ccd-420b-8b51-d7de76c27727_isrForCrosstalkSources_996716_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/cd29273a-3ccd-420b-8b51-d7de76c27727_isrForCrosstalkSources_996716_55.stderr parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 7ba5a8cb6afe285ca36f04e1f7dfaa7e parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c7439da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/01cda1a7-7543-4c05-936e-16ad545ef790_isrForCrosstalkSources_996712_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/01cda1a7-7543-4c05-936e-16ad545ef790_isrForCrosstalkSources_996712_34.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 81e99f8d55bc94b71c8f47dde22927f4 parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/35b72107-d3bb-4663-9fa1-0f11e2418c96_isrForCrosstalkSources_996706_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/35b72107-d3bb-4663-9fa1-0f11e2418c96_isrForCrosstalkSources_996706_26.stderr parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 32042121560647c8e4940c6a479ce7bd parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/635666da-1d42-44a4-b325-fcd4a20d1a87_isrForCrosstalkSources_996717_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/635666da-1d42-44a4-b325-fcd4a20d1a87_isrForCrosstalkSources_996717_56.stderr parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash 20e0bf8c7c97f7de0c329887fe390347 parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/678b4c40-3d08-4baf-bedc-2a1e0287aa30_isrForCrosstalkSources_996721_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/678b4c40-3d08-4baf-bedc-2a1e0287aa30_isrForCrosstalkSources_996721_33.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 02e79fb73c540aa364579ee5968568cf parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/32a20580-a695-4c5c-9c0e-cf1a5bf6d4d7_isrForCrosstalkSources_996707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/32a20580-a695-4c5c-9c0e-cf1a5bf6d4d7_isrForCrosstalkSources_996707_49.stderr parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash fdd74572ab2e1e86d0705378185aa964 parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/216ef912-bc2d-409b-af33-d57cc0761ae6_isrForCrosstalkSources_996717_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/216ef912-bc2d-409b-af33-d57cc0761ae6_isrForCrosstalkSources_996717_43.stderr parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 5382f047449b89665e78a3934853c1f3 parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f6447f4c-4b32-475e-aebd-16bfaa363499_isrForCrosstalkSources_996707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f6447f4c-4b32-475e-aebd-16bfaa363499_isrForCrosstalkSources_996707_36.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash 453d74230d8ebf34fab40170b0a58793 parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/767b73ac-0d97-4b18-a0c3-e0be436496ea_isrForCrosstalkSources_996717_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/767b73ac-0d97-4b18-a0c3-e0be436496ea_isrForCrosstalkSources_996717_49.stderr parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash ca952a59bb4cf2923482236242eeb29f parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/696f281e-e566-4b15-9fb1-38d8636d6709_isrForCrosstalkSources_996714_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/696f281e-e566-4b15-9fb1-38d8636d6709_isrForCrosstalkSources_996714_47.stderr parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash b174dee5ee6c204bde98068b77422ed6 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/b4c391bb-db3a-49e7-bc2d-a205494a7866_isrForCrosstalkSources_996708_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/b4c391bb-db3a-49e7-bc2d-a205494a7866_isrForCrosstalkSources_996708_43.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 7520630ea0c8a0e362b5ea714c104768 parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d10d7f2a-083c-40de-9ffa-80e74fd6a0a8_isrForCrosstalkSources_996709_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d10d7f2a-083c-40de-9ffa-80e74fd6a0a8_isrForCrosstalkSources_996709_8.stderr parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash 6693ce8c45a39ac786db08753417868e parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/7d6e247a-d477-43c3-82f3-ccf5a85e699a_isrForCrosstalkSources_996717_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/7d6e247a-d477-43c3-82f3-ccf5a85e699a_isrForCrosstalkSources_996717_22.stderr parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash 3cfb873c227e51e4315b5003812a343f parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 12:16:57 proc_lsst.multi:170 [INFO] [multi] job_id 17071139 proc_lsst.multi INFO: [multi] job_id 17071139 2024-03-15 12:16:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 12:16:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071139 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071139 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17071139 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-15 12:16:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-15 12:16:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 12:16:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f6f8f04f-7fe8-4b24-8d91-29dd0dc64e74_isrForCrosstalkSources_996717_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f6f8f04f-7fe8-4b24-8d91-29dd0dc64e74_isrForCrosstalkSources_996717_59.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash d3487684413e0b166458d84359d94b9b parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/88079fe0-733c-47ac-bba6-e151dbf24a52_isrForCrosstalkSources_996714_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/88079fe0-733c-47ac-bba6-e151dbf24a52_isrForCrosstalkSources_996714_36.stderr parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash 0a12ca0a3683d465c8493671ee8bf455 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/4c1a671e-d8e7-4b95-9bbe-e45ddba3eb1b_isrForCrosstalkSources_996706_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/4c1a671e-d8e7-4b95-9bbe-e45ddba3eb1b_isrForCrosstalkSources_996706_22.stderr parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 1d0fd0b10b11ae1e0dd6e6405af851dc parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a76c5c09-7189-4d05-8052-0c592a6ab786_isrForCrosstalkSources_996713_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a76c5c09-7189-4d05-8052-0c592a6ab786_isrForCrosstalkSources_996713_57.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash d84ecefce40af8120016987d109085d0 parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8ab194dd-3731-4ef9-bd50-d81e4325b7ed_isrForCrosstalkSources_996718_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8ab194dd-3731-4ef9-bd50-d81e4325b7ed_isrForCrosstalkSources_996718_29.stderr parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 713fd7e706c4262d0804c7e4a0079956 parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ac8e600b-922f-45fb-8e30-11fc157257d8_isrForCrosstalkSources_996716_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ac8e600b-922f-45fb-8e30-11fc157257d8_isrForCrosstalkSources_996716_30.stderr parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 7458c5972c59905c356c8e1d3bfad674 parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/e74eeafc-d052-49ea-9b6b-61f94222c051_isrForCrosstalkSources_996710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/e74eeafc-d052-49ea-9b6b-61f94222c051_isrForCrosstalkSources_996710_30.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash 0f2b18c0266e314dbc898fc26a53a5a6 parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/07ad7844-5d03-49da-9ff9-8d3147c7410b_isrForCrosstalkSources_996715_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/07ad7844-5d03-49da-9ff9-8d3147c7410b_isrForCrosstalkSources_996715_57.stderr parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash df7a118c14df43b0d0917d5373c9be15 parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/e539c47f-1af2-485d-9013-3e5c066a94bc_isrForCrosstalkSources_996714_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/e539c47f-1af2-485d-9013-3e5c066a94bc_isrForCrosstalkSources_996714_34.stderr parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 5d4f0fc374d5d6a84d35c951cbf10c60 parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/58676000-9240-405e-9eba-2bdcf70d8cd8_isrForCrosstalkSources_996710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/58676000-9240-405e-9eba-2bdcf70d8cd8_isrForCrosstalkSources_996710_32.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 24bc6668da0e41d5b75fab086fe9de89 parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f46507b7-9fb4-4e22-9f37-7998faaa9e90_isrForCrosstalkSources_996711_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f46507b7-9fb4-4e22-9f37-7998faaa9e90_isrForCrosstalkSources_996711_38.stderr parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash 172a40b3e6b66832202209190023ab2b parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/ec8c41cf-99fe-4db8-a5d2-cfc56b5ea9e9_isrForCrosstalkSources_996719_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/ec8c41cf-99fe-4db8-a5d2-cfc56b5ea9e9_isrForCrosstalkSources_996719_21.stderr parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash 1d43db0bbf4cb47d3a0dd2ed32529fea parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1fea4362-ffd1-43b6-abef-5fb1ae7b222c_isrForCrosstalkSources_996717_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1fea4362-ffd1-43b6-abef-5fb1ae7b222c_isrForCrosstalkSources_996717_33.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash 0f5fe7e27eb7e926ecdc4717310025c5 parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e30b2268-a3fc-454a-a626-0095cd6a6d61_isrForCrosstalkSources_996709_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e30b2268-a3fc-454a-a626-0095cd6a6d61_isrForCrosstalkSources_996709_24.stderr parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 1f5fa5cfeb089833148d95a0ec42ab3a parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/52a4b8dd-1aa0-48ed-b8a7-cd2b2f8ad86d_isrForCrosstalkSources_996720_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/52a4b8dd-1aa0-48ed-b8a7-cd2b2f8ad86d_isrForCrosstalkSources_996720_51.stderr parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash d12ed31994e79457194451968738c401 parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/31f8d234-d494-4d24-bdc5-354dac29a6be_isrForCrosstalkSources_996711_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/31f8d234-d494-4d24-bdc5-354dac29a6be_isrForCrosstalkSources_996711_52.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 9974014c1e6c6c72afe7d315b49a9c97 parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/b423151c-766c-46e6-aad5-2f886377288d_isrForCrosstalkSources_996710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/b423151c-766c-46e6-aad5-2f886377288d_isrForCrosstalkSources_996710_21.stderr parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 67fc92716b8f03d19613f648b91f45e7 parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/631779bd-0c0d-4d6a-9896-3010caa4b487_isrForCrosstalkSources_996721_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/631779bd-0c0d-4d6a-9896-3010caa4b487_isrForCrosstalkSources_996721_15.stderr parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash 7560f86d8b87b470efd5e6e1ed1bf5c9 parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/8eff4326-f3bc-40ad-bac8-36b41bb49264_isrForCrosstalkSources_996716_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/8eff4326-f3bc-40ad-bac8-36b41bb49264_isrForCrosstalkSources_996716_48.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash f12f28e66164f654b835ec5eba8710e2 parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4b06ea25-b112-4195-aa36-724e6105a337_isrForCrosstalkSources_996718_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4b06ea25-b112-4195-aa36-724e6105a337_isrForCrosstalkSources_996718_52.stderr parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash fc33e45f4c3c2053930403dfd4b3653c parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3ffc2d11-d11c-4987-85b2-439549a8c3af_isrForCrosstalkSources_996720_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3ffc2d11-d11c-4987-85b2-439549a8c3af_isrForCrosstalkSources_996720_52.stderr parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash e8c2678827797890355d23981b2ad8ab parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1508c743a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e15787a7-aa73-4e8f-9358-974d090d434a_isrForCrosstalkSources_996717_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e15787a7-aa73-4e8f-9358-974d090d434a_isrForCrosstalkSources_996717_55.stderr 2024-03-15 12:16:58 proc_lsst.multi:170 [INFO] [multi] job_id 17071141 proc_lsst.multi INFO: [multi] job_id 17071141 2024-03-15 12:16:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 12:16:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071141 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071141 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17071141 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-15 12:16:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-15 12:16:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 12:16:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:16:59 proc_lsst.multi:170 [INFO] [multi] job_id 17071142 proc_lsst.multi INFO: [multi] job_id 17071142 2024-03-15 12:16:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 12:16:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071142 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071142 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17071142 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-15 12:16:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-15 12:16:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:16:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:16:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:16:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:16:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 12:16:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:16:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:00 proc_lsst.multi:170 [INFO] [multi] job_id 17071143 proc_lsst.multi INFO: [multi] job_id 17071143 2024-03-15 12:17:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 12:17:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071143 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071143 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17071143 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-15 12:17:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-15 12:17:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 12:17:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab11f7f2da7efbbacf8ce1e9374de4e4 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/22c637b0-4093-4ab1-aa1f-7a2a9a84d26e_isrForCrosstalkSources_996711_9.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/22c637b0-4093-4ab1-aa1f-7a2a9a84d26e_isrForCrosstalkSources_996711_9.stderr 2024-03-15 12:17:01 proc_lsst.multi:170 [INFO] [multi] job_id 17071144 proc_lsst.multi INFO: [multi] job_id 17071144 2024-03-15 12:17:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 12:17:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071144 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071144 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17071144 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-15 12:17:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-15 12:17:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 12:17:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25a6bd3a7907fba404790c0c38f42fa2 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/42ca1f15-b6e3-4967-b80b-823aebb0c4b6_isrForCrosstalkSources_996712_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/42ca1f15-b6e3-4967-b80b-823aebb0c4b6_isrForCrosstalkSources_996712_51.stderr 2024-03-15 12:17:03 proc_lsst.multi:170 [INFO] [multi] job_id 17071148 proc_lsst.multi INFO: [multi] job_id 17071148 2024-03-15 12:17:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 12:17:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071148 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071148 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17071148 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-15 12:17:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-15 12:17:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 12:17:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:04 proc_lsst.multi:170 [INFO] [multi] job_id 17071149 proc_lsst.multi INFO: [multi] job_id 17071149 2024-03-15 12:17:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:17:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071149 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071149 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17071149 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-15 12:17:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-15 12:17:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:17:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:05 proc_lsst.multi:170 [INFO] [multi] job_id 17071151 proc_lsst.multi INFO: [multi] job_id 17071151 2024-03-15 12:17:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 12:17:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071151 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071151 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17071151 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-15 12:17:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-15 12:17:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 12:17:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:06 proc_lsst.multi:170 [INFO] [multi] job_id 17071152 proc_lsst.multi INFO: [multi] job_id 17071152 2024-03-15 12:17:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 12:17:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071152 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071152 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17071152 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-15 12:17:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-15 12:17:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 12:17:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:07 proc_lsst.multi:170 [INFO] [multi] job_id 17071153 proc_lsst.multi INFO: [multi] job_id 17071153 2024-03-15 12:17:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 12:17:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071153 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071153 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17071153 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-15 12:17:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-15 12:17:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 12:17:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65c1bfc9129d262747700b49e29cda2f with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/785c6e92-51f6-414b-a889-cdd0db11958e_isrForCrosstalkSources_996709_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/785c6e92-51f6-414b-a889-cdd0db11958e_isrForCrosstalkSources_996709_45.stderr 2024-03-15 12:17:08 proc_lsst.multi:170 [INFO] [multi] job_id 17071154 proc_lsst.multi INFO: [multi] job_id 17071154 2024-03-15 12:17:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 12:17:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071154 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071154 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17071154 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-15 12:17:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-15 12:17:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 12:17:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:08 proc_lsst.multi:170 [INFO] [multi] job_id 17071156 proc_lsst.multi INFO: [multi] job_id 17071156 2024-03-15 12:17:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 12:17:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071156 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071156 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17071156 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-15 12:17:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-15 12:17:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 12:17:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c46b853c386f8b57ebb27e5de562685 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/60e4f269-548c-4410-8e45-4ad10213276f_isrForCrosstalkSources_996719_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/60e4f269-548c-4410-8e45-4ad10213276f_isrForCrosstalkSources_996719_28.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c798c61e328e466fdbcbef794ff003b6 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/a8640b44-6478-49ff-b59e-8e25293847e5_isrForCrosstalkSources_996718_49.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/a8640b44-6478-49ff-b59e-8e25293847e5_isrForCrosstalkSources_996718_49.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3857c52296f212c71fdc30a7bf3d6d0e with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a923413f-f553-4d0a-97dc-98b60cd4478a_isrForCrosstalkSources_996708_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a923413f-f553-4d0a-97dc-98b60cd4478a_isrForCrosstalkSources_996708_35.stderr 2024-03-15 12:17:09 proc_lsst.multi:170 [INFO] [multi] job_id 17071157 proc_lsst.multi INFO: [multi] job_id 17071157 2024-03-15 12:17:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 12:17:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071157 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071157 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17071157 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-15 12:17:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-15 12:17:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 12:17:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 228861719e26daec27ba4b6958640425 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b67a555d-4e06-4375-ac9f-729e07ac942e_isrForCrosstalkSources_996705_21.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b67a555d-4e06-4375-ac9f-729e07ac942e_isrForCrosstalkSources_996705_21.stderr 2024-03-15 12:17:10 proc_lsst.multi:170 [INFO] [multi] job_id 17071158 proc_lsst.multi INFO: [multi] job_id 17071158 2024-03-15 12:17:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 12:17:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071158 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071158 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17071158 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-15 12:17:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-15 12:17:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 12:17:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:11 proc_lsst.multi:170 [INFO] [multi] job_id 17071160 proc_lsst.multi INFO: [multi] job_id 17071160 2024-03-15 12:17:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 12:17:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071160 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071160 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17071160 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-15 12:17:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-15 12:17:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 12:17:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97e55ac1a5dc75651d66835e815fe7ab with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8c8dd0c2-dd79-404f-b163-a9ed7d3fd579_isrForCrosstalkSources_996720_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8c8dd0c2-dd79-404f-b163-a9ed7d3fd579_isrForCrosstalkSources_996720_2.stderr 2024-03-15 12:17:12 proc_lsst.multi:170 [INFO] [multi] job_id 17071162 proc_lsst.multi INFO: [multi] job_id 17071162 2024-03-15 12:17:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 12:17:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071162 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071162 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17071162 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-15 12:17:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-15 12:17:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 12:17:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 238b9ff4f21d4763061c1daa2650bce3 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/bb54f20c-4ade-41e6-8d70-5fc4194430b0_isrForCrosstalkSources_996708_45.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/bb54f20c-4ade-41e6-8d70-5fc4194430b0_isrForCrosstalkSources_996708_45.stderr 2024-03-15 12:17:13 proc_lsst.multi:170 [INFO] [multi] job_id 17071163 proc_lsst.multi INFO: [multi] job_id 17071163 2024-03-15 12:17:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17071163 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17071163 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17071163 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-15 12:17:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-15 12:17:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 12:17:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2552b9fa08bce76c5c9086e640989a4 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/40371183-530d-4b77-8c85-b4afacc2e445_isrForCrosstalkSources_996720_57.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/40371183-530d-4b77-8c85-b4afacc2e445_isrForCrosstalkSources_996720_57.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d577818519ba34e5657c482fb3f84d5 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/fd9a92dc-acd4-4d13-a3c8-45790338273e_isrForCrosstalkSources_996712_57.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/fd9a92dc-acd4-4d13-a3c8-45790338273e_isrForCrosstalkSources_996712_57.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23e9a8feaf3855a98cd35c0a4826e5e5 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/20581659-8f15-48ae-9182-abe711a5128d_isrForCrosstalkSources_996711_58.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/20581659-8f15-48ae-9182-abe711a5128d_isrForCrosstalkSources_996711_58.stderr 2024-03-15 12:17:14 proc_lsst.multi:170 [INFO] [multi] job_id 17071164 proc_lsst.multi INFO: [multi] job_id 17071164 2024-03-15 12:17:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:14 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17071164 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17071164 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17071164 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-15 12:17:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-15 12:17:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfcdc49d7b1c123c62255e405d37c94f with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/febf64d0-619f-49f7-92e7-6b24a97c601f_isrForCrosstalkSources_996719_18.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/febf64d0-619f-49f7-92e7-6b24a97c601f_isrForCrosstalkSources_996719_18.stderr 2024-03-15 12:17:15 proc_lsst.multi:170 [INFO] [multi] job_id 17071165 proc_lsst.multi INFO: [multi] job_id 17071165 2024-03-15 12:17:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 12:17:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:15 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17071165 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17071165 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17071165 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-15 12:17:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-15 12:17:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 12:17:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:16 proc_lsst.multi:170 [INFO] [multi] job_id 17071166 proc_lsst.multi INFO: [multi] job_id 17071166 2024-03-15 12:17:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 12:17:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:16 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17071166 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17071166 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17071166 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-15 12:17:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-15 12:17:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 12:17:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:18 proc_lsst.multi:170 [INFO] [multi] job_id 17071199 proc_lsst.multi INFO: [multi] job_id 17071199 2024-03-15 12:17:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 12:17:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:18 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17071199 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17071199 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17071199 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23127545334096 2024-03-15 12:17:18 proc_lsst.multi:146 [INFO] found job 91474 in provider local proc_lsst.multi INFO: found job 91474 in provider local 2024-03-15 12:17:18 proc_lsst.multi:146 [INFO] found job 17071124 in provider astro proc_lsst.multi INFO: found job 17071124 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:20 proc_lsst.multi:146 [INFO] found job 17071125 in provider astro proc_lsst.multi INFO: found job 17071125 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 967bf5730ce3df38d976267f645fb323 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7c71db36-1ebe-4984-a1db-3c5ad3b62bf7_isrForCrosstalkSources_996712_23.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7c71db36-1ebe-4984-a1db-3c5ad3b62bf7_isrForCrosstalkSources_996712_23.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9439381096e552cf488b4e90a010616 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ec927082-69e6-4b23-91cc-a21f479556d5_isrForCrosstalkSources_996713_11.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ec927082-69e6-4b23-91cc-a21f479556d5_isrForCrosstalkSources_996713_11.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22de3910ecc45ded504311a84e9c11e0 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8ab25203-301e-409c-a314-c84dc0ccbb34_isrForCrosstalkSources_996720_28.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8ab25203-301e-409c-a314-c84dc0ccbb34_isrForCrosstalkSources_996720_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:21 proc_lsst.multi:146 [INFO] found job 17071126 in provider astro proc_lsst.multi INFO: found job 17071126 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 868bd0e1f6581694a51e3edfe66764d9 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5a240a42-0fd2-4d55-9d62-b19465144d78_isrForCrosstalkSources_996707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5a240a42-0fd2-4d55-9d62-b19465144d78_isrForCrosstalkSources_996707_48.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 599b75b8263f3e24ee2ed768ae27e203 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c767a7e2-e490-48d2-9895-6ceeeb517165_isrForCrosstalkSources_996709_61.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c767a7e2-e490-48d2-9895-6ceeeb517165_isrForCrosstalkSources_996709_61.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b371c3702226430405c4548e8a3528c5 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/03a7886a-09d1-4b97-94cb-6878b1b6537f_isrForCrosstalkSources_996713_38.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/03a7886a-09d1-4b97-94cb-6878b1b6537f_isrForCrosstalkSources_996713_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:23 proc_lsst.multi:146 [INFO] found job 17071127 in provider astro proc_lsst.multi INFO: found job 17071127 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:24 proc_lsst.multi:146 [INFO] found job 17071128 in provider astro proc_lsst.multi INFO: found job 17071128 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6180900d07d5b81ad434a615c09d98f9 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0bf27885-92a0-43b5-9d56-312b2ae5ba95_isrForCrosstalkSources_996715_53.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0bf27885-92a0-43b5-9d56-312b2ae5ba95_isrForCrosstalkSources_996715_53.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06a9414cf848f518af7b354f63f8cfdf with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e82e41f6-0aaf-4515-aa68-f6004f80e760_isrForCrosstalkSources_996715_14.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e82e41f6-0aaf-4515-aa68-f6004f80e760_isrForCrosstalkSources_996715_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:25 proc_lsst.multi:146 [INFO] found job 17071130 in provider astro proc_lsst.multi INFO: found job 17071130 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff91e3e98195b1bffb1e562be81d8d0d with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a05be6ae-3d0e-403c-a294-106c7edb763a_isrForCrosstalkSources_996709_6.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a05be6ae-3d0e-403c-a294-106c7edb763a_isrForCrosstalkSources_996709_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:26 proc_lsst.multi:146 [INFO] found job 17071131 in provider astro proc_lsst.multi INFO: found job 17071131 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry add675783ad690103f0294b5d2326e2f with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fb89f2c1-2204-47e4-8d86-c7086b6a58e6_isrForCrosstalkSources_996721_44.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fb89f2c1-2204-47e4-8d86-c7086b6a58e6_isrForCrosstalkSources_996721_44.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17aae9bcb56e9850bbd9c2055e0bf23a with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/820f2e1d-d35e-4f6e-8aae-3d5c4d4f94b4_isrForCrosstalkSources_996708_41.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/820f2e1d-d35e-4f6e-8aae-3d5c4d4f94b4_isrForCrosstalkSources_996708_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:26 proc_lsst.multi:146 [INFO] found job 17071132 in provider astro proc_lsst.multi INFO: found job 17071132 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:27 proc_lsst.multi:146 [INFO] found job 17071133 in provider astro proc_lsst.multi INFO: found job 17071133 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:28 proc_lsst.multi:146 [INFO] found job 17071134 in provider astro proc_lsst.multi INFO: found job 17071134 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 054bf1a9bf4c3bfb6b88e8b7b635b524 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/59a20509-e73c-43d8-89c5-0ebc795ca147_isrForCrosstalkSources_996706_12.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/59a20509-e73c-43d8-89c5-0ebc795ca147_isrForCrosstalkSources_996706_12.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8341475b7232a95aed56d6990271e9fd with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1485fe1b-ebc9-41b7-90ff-2e56a3024261_isrForCrosstalkSources_996717_60.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1485fe1b-ebc9-41b7-90ff-2e56a3024261_isrForCrosstalkSources_996717_60.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecb368be92f6e9e7f98877380c321d50 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e6febafe-78b3-4fa1-8299-42b75a60b2db_isrForCrosstalkSources_996706_11.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e6febafe-78b3-4fa1-8299-42b75a60b2db_isrForCrosstalkSources_996706_11.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85aafd8b08da3bbeeae033105a90803e with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/1dc1c8d3-282f-4d9f-918b-99c26eacc31b_isrForCrosstalkSources_996715_8.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/1dc1c8d3-282f-4d9f-918b-99c26eacc31b_isrForCrosstalkSources_996715_8.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f9a91e9fe2c9fb664c55995a148861f with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/2a42eb7d-2b38-49e2-bb1b-7bc7fbd8892e_isrForCrosstalkSources_996708_56.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/2a42eb7d-2b38-49e2-bb1b-7bc7fbd8892e_isrForCrosstalkSources_996708_56.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2210435886f0b785277502efe8f1bee with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/671059d6-d4dc-4a60-bc00-dde4bcb17531_isrForCrosstalkSources_996717_18.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/671059d6-d4dc-4a60-bc00-dde4bcb17531_isrForCrosstalkSources_996717_18.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea7323d872eb3ffa7e50459c9d15a8bf with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a475474c-d189-452c-ba1e-5473e2c55e8b_isrForCrosstalkSources_996708_57.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a475474c-d189-452c-ba1e-5473e2c55e8b_isrForCrosstalkSources_996708_57.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2752735237b8eee238e8fc9bd4020bb9 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f1bba586-e963-4c07-85c5-b445652ae7ca_isrForCrosstalkSources_996717_2.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f1bba586-e963-4c07-85c5-b445652ae7ca_isrForCrosstalkSources_996717_2.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cffb766a6a55635bbc2d4caa89324df4 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/09f557f9-3106-4cc9-930f-9285432821d6_isrForCrosstalkSources_996718_15.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/09f557f9-3106-4cc9-930f-9285432821d6_isrForCrosstalkSources_996718_15.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86a6d2c00772a3338cc552585797d1c9 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2e76b13c-f556-4ba9-9346-17b55ec2fad4_isrForCrosstalkSources_996710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2e76b13c-f556-4ba9-9346-17b55ec2fad4_isrForCrosstalkSources_996710_46.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba3f4bc13654f2a92725e64b579681ed with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8b3831c5-f98e-402c-bc97-0df0f3ac4e9a_isrForCrosstalkSources_996712_9.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8b3831c5-f98e-402c-bc97-0df0f3ac4e9a_isrForCrosstalkSources_996712_9.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc202907308913389c42dc1f595c81b7 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/864a25b7-d53e-4040-b646-cd2afece6e1c_isrForCrosstalkSources_996711_4.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/864a25b7-d53e-4040-b646-cd2afece6e1c_isrForCrosstalkSources_996711_4.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e9774bb115c883bbf9123cc2d712532 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/fc7bd063-b5ce-4455-8884-a5f3d599c53c_isrForCrosstalkSources_996711_17.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/fc7bd063-b5ce-4455-8884-a5f3d599c53c_isrForCrosstalkSources_996711_17.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0757124bef9b1d82674d65568047b98 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/a3a70161-e716-47a2-bb7a-23f8d9c5caef_isrForCrosstalkSources_996718_20.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/a3a70161-e716-47a2-bb7a-23f8d9c5caef_isrForCrosstalkSources_996718_20.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81cc9c1ae5cc89b160cabd8f6b1901e7 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/38fc74a6-5303-4196-8cc0-eb34ad22fb6b_isrForCrosstalkSources_996716_36.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/38fc74a6-5303-4196-8cc0-eb34ad22fb6b_isrForCrosstalkSources_996716_36.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68ebe68c563e9fbc824892bc697094ae with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1474773d-fd6e-4279-ba3c-f9fccfa31c75_isrForCrosstalkSources_996721_56.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1474773d-fd6e-4279-ba3c-f9fccfa31c75_isrForCrosstalkSources_996721_56.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d3b489a3f39d9885eedd16df30eb332 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/34ec7e42-97b5-41cd-8602-289f94f2f1cd_isrForCrosstalkSources_996710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/34ec7e42-97b5-41cd-8602-289f94f2f1cd_isrForCrosstalkSources_996710_54.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67bef6f2a8f9aa0c48923bb68ee9c388 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/88a30682-77e9-42b0-964f-2b2bd0a6ea82_isrForCrosstalkSources_996707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/88a30682-77e9-42b0-964f-2b2bd0a6ea82_isrForCrosstalkSources_996707_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:30 proc_lsst.multi:146 [INFO] found job 17071135 in provider astro proc_lsst.multi INFO: found job 17071135 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3bcbee7a3e6b1ee3cea0a72e094cccf with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c8e38166-f0b0-4be6-a5fb-956186b1b8e7_isrForCrosstalkSources_996709_2.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c8e38166-f0b0-4be6-a5fb-956186b1b8e7_isrForCrosstalkSources_996709_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:31 proc_lsst.multi:146 [INFO] found job 17071136 in provider astro proc_lsst.multi INFO: found job 17071136 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:32 proc_lsst.multi:146 [INFO] found job 17071137 in provider astro proc_lsst.multi INFO: found job 17071137 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:33 proc_lsst.multi:146 [INFO] found job 17071139 in provider astro proc_lsst.multi INFO: found job 17071139 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:34 proc_lsst.multi:146 [INFO] found job 17071141 in provider astro proc_lsst.multi INFO: found job 17071141 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:36 proc_lsst.multi:146 [INFO] found job 17071142 in provider astro proc_lsst.multi INFO: found job 17071142 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7862396dc93c92b7e007be0d65eefe06 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/59d9e6e2-5d1c-4fde-8729-c49f7c6f314c_isrForCrosstalkSources_996720_15.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/59d9e6e2-5d1c-4fde-8729-c49f7c6f314c_isrForCrosstalkSources_996720_15.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6292d9fd492407841d394a0e115d78dd with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f49d01df-024f-48ea-8818-755182c5f8cb_isrForCrosstalkSources_996712_17.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f49d01df-024f-48ea-8818-755182c5f8cb_isrForCrosstalkSources_996712_17.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 449d0e9a787a1adc88d6553fbf984ddd with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c67d0c4b-7848-4186-866d-6948be12cbec_isrForCrosstalkSources_996713_17.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c67d0c4b-7848-4186-866d-6948be12cbec_isrForCrosstalkSources_996713_17.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7b99b6472290b4d24bbdfa122a7721 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/06b03eec-8a0a-45e2-bcaf-8f0779f93943_isrForCrosstalkSources_996717_6.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/06b03eec-8a0a-45e2-bcaf-8f0779f93943_isrForCrosstalkSources_996717_6.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca7dc4a6e8e42a242fdc14f851148364 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/7c86ce70-adce-4382-9082-0ecc42e02c26_isrForCrosstalkSources_996717_37.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/7c86ce70-adce-4382-9082-0ecc42e02c26_isrForCrosstalkSources_996717_37.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86f813e4633ff3568f3ab8827756d4db with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/be7242a8-5d5b-4740-a1f2-1e34f9cd0632_isrForCrosstalkSources_996711_26.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/be7242a8-5d5b-4740-a1f2-1e34f9cd0632_isrForCrosstalkSources_996711_26.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d223a63bc9faa1186548dbfa2a72573 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/398ce131-9e84-42f9-a75c-78c36e264884_isrForCrosstalkSources_996707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/398ce131-9e84-42f9-a75c-78c36e264884_isrForCrosstalkSources_996707_2.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63b17e066c1a54221fcb0d76489712c1 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5e078cc3-823b-4903-84b6-967e1ef95827_isrForCrosstalkSources_996719_43.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5e078cc3-823b-4903-84b6-967e1ef95827_isrForCrosstalkSources_996719_43.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c895cd4b25f1f83d6e37f4bd752ef13 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/00a7d73d-ecfd-4fad-8b48-f030395abaf2_isrForCrosstalkSources_996715_29.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/00a7d73d-ecfd-4fad-8b48-f030395abaf2_isrForCrosstalkSources_996715_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:37 proc_lsst.multi:146 [INFO] found job 17071143 in provider astro proc_lsst.multi INFO: found job 17071143 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75b4c8eb25f8540f87f651784fed78f0 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/aa8ddcfe-c93f-404a-a7c0-3d21621a3dde_isrForCrosstalkSources_996710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/aa8ddcfe-c93f-404a-a7c0-3d21621a3dde_isrForCrosstalkSources_996710_19.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adb0be5a9fc083f84d607f1cc1f89e52 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/95469df3-3b50-4abe-8a11-465438a7824f_isrForCrosstalkSources_996713_18.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/95469df3-3b50-4abe-8a11-465438a7824f_isrForCrosstalkSources_996713_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry cc99fcd8206f92bfc8a7458a7b932ca9 with result from task 57 parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:38 proc_lsst.multi:146 [INFO] found job 17071144 in provider astro proc_lsst.multi INFO: found job 17071144 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f8c46768-a67c-44a1-896f-ec2a59136319_isrForCrosstalkSources_996714_57.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f8c46768-a67c-44a1-896f-ec2a59136319_isrForCrosstalkSources_996714_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:39 proc_lsst.multi:146 [INFO] found job 17071148 in provider astro proc_lsst.multi INFO: found job 17071148 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:39 proc_lsst.multi:146 [INFO] found job 17071149 in provider astro proc_lsst.multi INFO: found job 17071149 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d860cf4dea43cf8b829558127a8eb6e with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a5e0f51f-0681-4811-9e76-0e787a0f67bd_isrForCrosstalkSources_996708_27.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a5e0f51f-0681-4811-9e76-0e787a0f67bd_isrForCrosstalkSources_996708_27.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28f6e23fe3220572cbc7f7ea10518ef3 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/df220403-c22c-4046-8c3e-52d16d0545fd_isrForCrosstalkSources_996707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/df220403-c22c-4046-8c3e-52d16d0545fd_isrForCrosstalkSources_996707_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:40 proc_lsst.multi:146 [INFO] found job 17071151 in provider astro proc_lsst.multi INFO: found job 17071151 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fc2372c00b78961fa8efbd619fef3ac with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/9f8edd92-987a-426d-882e-49e46000dcd7_isrForCrosstalkSources_996712_40.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/9f8edd92-987a-426d-882e-49e46000dcd7_isrForCrosstalkSources_996712_40.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88cfeeac8f0be3e7de2b7201f8862796 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/37bf3ecc-8b9c-4274-91cd-1c47a3fd7cc7_isrForCrosstalkSources_996708_4.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/37bf3ecc-8b9c-4274-91cd-1c47a3fd7cc7_isrForCrosstalkSources_996708_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:41 proc_lsst.multi:146 [INFO] found job 17071152 in provider astro proc_lsst.multi INFO: found job 17071152 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09fdd6e6c8b90fce8c117584069ae3d8 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cbc19148-dece-4155-990a-bfc23c9b858a_isrForCrosstalkSources_996714_23.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cbc19148-dece-4155-990a-bfc23c9b858a_isrForCrosstalkSources_996714_23.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 065084dbee8bb7d51e0b273c01a7f3ee with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/eb19f250-f8fa-444b-9ad4-3f82498aa5d9_isrForCrosstalkSources_996716_45.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/eb19f250-f8fa-444b-9ad4-3f82498aa5d9_isrForCrosstalkSources_996716_45.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8c4f98285899a2fcbd50a19e64fb7a with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/4fbc94d2-e6f6-4ba2-83fc-95695ec2f334_isrForCrosstalkSources_996709_40.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/4fbc94d2-e6f6-4ba2-83fc-95695ec2f334_isrForCrosstalkSources_996709_40.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8bb7a886340246a71c53bc8bbc98cd1 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/073ff42b-9f55-43e9-b178-cae77a06591b_isrForCrosstalkSources_996710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/073ff42b-9f55-43e9-b178-cae77a06591b_isrForCrosstalkSources_996710_3.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6bceaa03dcf85a48dbe72c3323ab1b7 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d861bed1-8631-4184-bb2b-3e135ca9fc16_isrForCrosstalkSources_996705_20.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d861bed1-8631-4184-bb2b-3e135ca9fc16_isrForCrosstalkSources_996705_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:41 proc_lsst.multi:146 [INFO] found job 17071153 in provider astro proc_lsst.multi INFO: found job 17071153 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a383aa6ae71d41779a1cc1af2ab6b8d with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/445eeb27-3d1d-4b90-8c14-f86584e6ebe0_isrForCrosstalkSources_996719_51.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/445eeb27-3d1d-4b90-8c14-f86584e6ebe0_isrForCrosstalkSources_996719_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:42 proc_lsst.multi:146 [INFO] found job 17071154 in provider astro proc_lsst.multi INFO: found job 17071154 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72f7739b01aed8d82655a652d06d7c4b with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/494067ae-c56e-4f9c-9c95-5b3db181c384_isrForCrosstalkSources_996708_30.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/494067ae-c56e-4f9c-9c95-5b3db181c384_isrForCrosstalkSources_996708_30.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d54aa2e87a98c995c1e7af2cfd33005 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/346d0b78-19b5-4d85-9f0b-ddd97cf5cc42_isrForCrosstalkSources_996719_37.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/346d0b78-19b5-4d85-9f0b-ddd97cf5cc42_isrForCrosstalkSources_996719_37.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11f8aaecf3a7e8975a9dec188b9c89b1 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c23cf8bd-98bd-4509-9516-ee28360ae7fa_isrForCrosstalkSources_996719_49.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c23cf8bd-98bd-4509-9516-ee28360ae7fa_isrForCrosstalkSources_996719_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:42 proc_lsst.multi:146 [INFO] found job 17071156 in provider astro proc_lsst.multi INFO: found job 17071156 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:43 proc_lsst.multi:146 [INFO] found job 17071157 in provider astro proc_lsst.multi INFO: found job 17071157 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a209da0a3bc4a3a33e28fbd3b4883ff with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1595f95f-1038-4639-9618-bf267c83ceb3_isrForCrosstalkSources_996712_7.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1595f95f-1038-4639-9618-bf267c83ceb3_isrForCrosstalkSources_996712_7.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b80b39ca7b0f6809a66a139e633e7b7 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/7363aeec-2d50-447d-956f-8af167cc345a_isrForCrosstalkSources_996707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/7363aeec-2d50-447d-956f-8af167cc345a_isrForCrosstalkSources_996707_34.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed6f831036e5af3e0e534469e65cb1bf with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/33bfa71b-c976-4cd9-8767-8297ae874987_isrForCrosstalkSources_996721_19.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/33bfa71b-c976-4cd9-8767-8297ae874987_isrForCrosstalkSources_996721_19.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7528595378b332dbcd95dd19c5fa8afb with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/9b5e459a-0bce-47a9-b5b1-15390d0f22ff_isrForCrosstalkSources_996720_42.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/9b5e459a-0bce-47a9-b5b1-15390d0f22ff_isrForCrosstalkSources_996720_42.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afc304e1ccaa4f3b692671501cbef46f with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/5b976ab3-0742-46eb-bef7-ca57c55020fc_isrForCrosstalkSources_996714_53.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/5b976ab3-0742-46eb-bef7-ca57c55020fc_isrForCrosstalkSources_996714_53.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b00471d568c62a78f75f832abe79e28 with result from task 54 parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:43 proc_lsst.multi:146 [INFO] found job 17071158 in provider astro proc_lsst.multi INFO: found job 17071158 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/44324103-6c16-4aeb-bd58-ac85571ff8f3_isrForCrosstalkSources_996721_61.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/44324103-6c16-4aeb-bd58-ac85571ff8f3_isrForCrosstalkSources_996721_61.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87a993310dd56bdf40755e4a178d7f18 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/14b046d2-e760-4942-89a8-bf79ca54065c_isrForCrosstalkSources_996709_57.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/14b046d2-e760-4942-89a8-bf79ca54065c_isrForCrosstalkSources_996709_57.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8558d8d0f45d823f5d8ea8bc876fcd74 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5a0e0fed-bbd1-4c70-80da-bbcfa75c54d9_isrForCrosstalkSources_996721_58.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5a0e0fed-bbd1-4c70-80da-bbcfa75c54d9_isrForCrosstalkSources_996721_58.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1a91aa200ec56cbdc08e968bdf601e with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/f731dd4c-e4cd-4f0d-aac9-df451eaab7d3_isrForCrosstalkSources_996715_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/f731dd4c-e4cd-4f0d-aac9-df451eaab7d3_isrForCrosstalkSources_996715_58.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a07d2ccff0ede33bab699bf85851ca07 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/da3bf7ba-176b-4e6e-8090-afacf281a4c9_isrForCrosstalkSources_996715_48.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/da3bf7ba-176b-4e6e-8090-afacf281a4c9_isrForCrosstalkSources_996715_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:45 proc_lsst.multi:146 [INFO] found job 17071160 in provider astro proc_lsst.multi INFO: found job 17071160 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f518c4939f0a68bd48700a315ca28c6 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b46a184a-80df-427e-81d0-2dacb2113c43_isrForCrosstalkSources_996715_17.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b46a184a-80df-427e-81d0-2dacb2113c43_isrForCrosstalkSources_996715_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:46 proc_lsst.multi:146 [INFO] found job 17071162 in provider astro proc_lsst.multi INFO: found job 17071162 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:46 proc_lsst.multi:146 [INFO] found job 17071163 in provider astro proc_lsst.multi INFO: found job 17071163 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071124,17071125,17071126,17071127,17071128,17071130,17071131,17071132,17071133,17071134,17071135,17071136,17071137,17071139,17071141,17071142,17071143,17071144,17071148,17071149,17071151,17071152,17071153,17071154,17071156,17071157,17071158,17071160,17071162,17071163' parsl.providers.slurm.slurm DEBUG: squeue returned 17071163 R 17071162 R 17071160 R 17071156 R 17071157 R 17071158 R 17071152 R 17071153 R 17071154 R 17071148 R 17071149 R 17071151 R 17071142 R 17071143 R 17071144 R 17071137 R 17071139 R 17071141 R 17071134 R 17071135 R 17071136 R 17071130 R 17071131 R 17071132 R 17071133 R 17071125 R 17071126 R 17071127 R 17071128 R 17071124 R parsl.providers.slurm.slurm DEBUG: Updating job 17071163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:47 proc_lsst.multi:146 [INFO] found job 17071164 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071164 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071164,17071165,17071166,17071199' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e14783cfa992364bd8c4d52188e0d980 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d4052aea-ea8b-4437-a97c-f3c00189a6b2_isrForCrosstalkSources_996713_29.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d4052aea-ea8b-4437-a97c-f3c00189a6b2_isrForCrosstalkSources_996713_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071199 R 17071165 R 17071166 R 17071164 R parsl.providers.slurm.slurm DEBUG: Updating job 17071199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071164 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:49 proc_lsst.multi:146 [INFO] found job 17071165 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071165 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071164,17071165,17071166,17071199' parsl.providers.slurm.slurm DEBUG: squeue returned 17071199 R 17071165 R 17071166 R 17071164 R parsl.providers.slurm.slurm DEBUG: Updating job 17071199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071164 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:51 proc_lsst.multi:146 [INFO] found job 17071166 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071166 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071164,17071165,17071166,17071199' parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 702bc342fa6240bed5c03fac5bbb77ae with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/26be4ed8-fe6e-492c-ae7a-e1966c28c722_isrForCrosstalkSources_996710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/26be4ed8-fe6e-492c-ae7a-e1966c28c722_isrForCrosstalkSources_996710_26.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8179380bcd7ee910ea9651ce7cb213d0 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/9874d830-704b-4481-a996-3a128a5cbc54_isrForCrosstalkSources_996719_46.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/9874d830-704b-4481-a996-3a128a5cbc54_isrForCrosstalkSources_996719_46.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c67704aa516056ee74d10e3e9b5c943 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e51e5685-7ef3-4ae5-8a58-ae57287b292b_isrForCrosstalkSources_996706_13.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e51e5685-7ef3-4ae5-8a58-ae57287b292b_isrForCrosstalkSources_996706_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071199 R 17071165 R 17071166 R 17071164 R parsl.providers.slurm.slurm DEBUG: Updating job 17071199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071164 with slurm status R to parsl state JobState.RUNNING 2024-03-15 12:17:52 proc_lsst.multi:146 [INFO] found job 17071199 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071199 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17071164,17071165,17071166,17071199' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0741f5719b721ac55507178c2c5f9823 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/3c1b5a31-bc86-4b1f-915c-b0b0d03b2786_isrForCrosstalkSources_996714_3.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/3c1b5a31-bc86-4b1f-915c-b0b0d03b2786_isrForCrosstalkSources_996714_3.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c67ad129da569a8d8e7645a0f85fdad6 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/73964f24-fb0f-4f32-8556-38f64e21fed7_isrForCrosstalkSources_996718_51.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/73964f24-fb0f-4f32-8556-38f64e21fed7_isrForCrosstalkSources_996718_51.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61b047fe3a930382ea7e5b1cb23a6ac3 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2e8501f7-d32d-4e20-950b-fdde51ef6784_isrForCrosstalkSources_996718_11.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2e8501f7-d32d-4e20-950b-fdde51ef6784_isrForCrosstalkSources_996718_11.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51a76585669a2a7d1404c945b14ff875 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/9b39daf5-9aa9-44c7-adb5-fe4166658a2c_isrForCrosstalkSources_996711_6.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/9b39daf5-9aa9-44c7-adb5-fe4166658a2c_isrForCrosstalkSources_996711_6.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 729d2081d3f6ffaaf3c77e01d210f1ba with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6df6a1c9-bb99-4e18-a679-b2c9231747b4_isrForCrosstalkSources_996713_42.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6df6a1c9-bb99-4e18-a679-b2c9231747b4_isrForCrosstalkSources_996713_42.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcca447f85364ea9990a74b2a248ffdf with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/986c3a69-3787-4295-ad15-89d4674cdbbc_isrForCrosstalkSources_996708_11.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/986c3a69-3787-4295-ad15-89d4674cdbbc_isrForCrosstalkSources_996708_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17071199 R 17071165 R 17071166 R 17071164 R parsl.providers.slurm.slurm DEBUG: Updating job 17071199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17071164 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c45832c5375f9270d08026d0d068fcb with result from task 82 parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 35, active_tasks = 964 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.strategy DEBUG: Executor multi has 964 active tasks, 35/0 running/pending blocks, and 36 connected workers parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/dfebe05f-7c15-4856-8899-168498555da3_isrForCrosstalkSources_996721_43.stdout parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/dfebe05f-7c15-4856-8899-168498555da3_isrForCrosstalkSources_996721_43.stderr parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 35 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 206 more blocks parsl.executors.status_handling INFO: Scaling out by 206 blocks parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-15 12:17:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-15 12:17:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 12:17:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b10192320a0e07866e58ea2df38982ca with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d9473cda-25dd-4845-b55a-391963c1462f_isrForCrosstalkSources_996705_50.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d9473cda-25dd-4845-b55a-391963c1462f_isrForCrosstalkSources_996705_50.stderr 2024-03-15 12:17:53 proc_lsst.multi:170 [INFO] [multi] job_id 17071208 proc_lsst.multi INFO: [multi] job_id 17071208 2024-03-15 12:17:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:53 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17071208 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17071208 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17071208 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-15 12:17:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-15 12:17:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 12:17:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:54 proc_lsst.multi:170 [INFO] [multi] job_id 17071209 proc_lsst.multi INFO: [multi] job_id 17071209 2024-03-15 12:17:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:54 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071209 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071209 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17071209 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-15 12:17:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-15 12:17:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6540bb06f01453f6aa66f2cc9fbbf1d with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/603c8710-f0ff-4eae-bc1e-b0d8c33d9aff_isrForCrosstalkSources_996715_33.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/603c8710-f0ff-4eae-bc1e-b0d8c33d9aff_isrForCrosstalkSources_996715_33.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cfaf4ac19c3e10fc5fbdda9de5a05bc with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5874f131-6109-42d1-8b5a-0116ec1a1a03_isrForCrosstalkSources_996710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5874f131-6109-42d1-8b5a-0116ec1a1a03_isrForCrosstalkSources_996710_22.stderr 2024-03-15 12:17:54 proc_lsst.multi:170 [INFO] [multi] job_id 17071210 proc_lsst.multi INFO: [multi] job_id 17071210 2024-03-15 12:17:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 12:17:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:54 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071210 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071210 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17071210 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-15 12:17:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-15 12:17:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 12:17:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 287d392c3f89ceaec2869a61bfb5b6c8 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/3151793c-3f33-4ed4-a1be-99a7266b09f2_isrForCrosstalkSources_996713_13.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/3151793c-3f33-4ed4-a1be-99a7266b09f2_isrForCrosstalkSources_996713_13.stderr 2024-03-15 12:17:55 proc_lsst.multi:170 [INFO] [multi] job_id 17071211 proc_lsst.multi INFO: [multi] job_id 17071211 2024-03-15 12:17:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 12:17:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:55 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071211 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071211 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17071211 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-15 12:17:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-15 12:17:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 12:17:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:55 proc_lsst.multi:170 [INFO] [multi] job_id 17071212 proc_lsst.multi INFO: [multi] job_id 17071212 2024-03-15 12:17:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 12:17:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:55 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071212 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071212 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17071212 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-15 12:17:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-15 12:17:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 12:17:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6352e3bc3ace1816b2d4846535e1ea92 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/19091f73-8361-4c02-bbd5-6e53116b20a3_isrForCrosstalkSources_996718_24.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/19091f73-8361-4c02-bbd5-6e53116b20a3_isrForCrosstalkSources_996718_24.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 537654709649c7583dc31264da3b31d5 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed905e3c-97de-443a-9538-077c342866a4_isrForCrosstalkSources_996709_22.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed905e3c-97de-443a-9538-077c342866a4_isrForCrosstalkSources_996709_22.stderr 2024-03-15 12:17:56 proc_lsst.multi:170 [INFO] [multi] job_id 17071213 proc_lsst.multi INFO: [multi] job_id 17071213 2024-03-15 12:17:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071213 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071213 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17071213 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-15 12:17:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-15 12:17:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b529dd16e935356874dd973887a230c9 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/d9dac860-9a7f-4f45-b5d6-f0afb5ecee4a_isrForCrosstalkSources_996710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/d9dac860-9a7f-4f45-b5d6-f0afb5ecee4a_isrForCrosstalkSources_996710_2.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d7471266af3ca448f41cdf2a9a645e0 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1a2a9ce3-a21e-448d-8dad-8eda983f2445_isrForCrosstalkSources_996707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1a2a9ce3-a21e-448d-8dad-8eda983f2445_isrForCrosstalkSources_996707_30.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06272fc8eb3c7a915217cca54e94256b with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a6c49934-f172-499a-b534-84adf6f94baf_isrForCrosstalkSources_996706_24.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a6c49934-f172-499a-b534-84adf6f94baf_isrForCrosstalkSources_996706_24.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adb00d19cb609536d636399d401cdd5e with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/641acaba-587f-49be-b1f6-f7af92190a90_isrForCrosstalkSources_996718_17.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/641acaba-587f-49be-b1f6-f7af92190a90_isrForCrosstalkSources_996718_17.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 523f1e02327d16b530b1b27ba368a606 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/47a83f36-5d89-4d7c-ae8d-1750a2c91cb7_isrForCrosstalkSources_996705_30.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/47a83f36-5d89-4d7c-ae8d-1750a2c91cb7_isrForCrosstalkSources_996705_30.stderr 2024-03-15 12:17:56 proc_lsst.multi:170 [INFO] [multi] job_id 17071214 proc_lsst.multi INFO: [multi] job_id 17071214 2024-03-15 12:17:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 12:17:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071214 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071214 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17071214 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-15 12:17:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-15 12:17:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 12:17:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:57 proc_lsst.multi:170 [INFO] [multi] job_id 17071215 proc_lsst.multi INFO: [multi] job_id 17071215 2024-03-15 12:17:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 12:17:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071215 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071215 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17071215 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-15 12:17:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-15 12:17:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 12:17:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30994fec4f238ed15522ffb299f9064e with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/e7c0be03-c22d-4808-8bf1-d1c271ee4039_isrForCrosstalkSources_996712_18.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/e7c0be03-c22d-4808-8bf1-d1c271ee4039_isrForCrosstalkSources_996712_18.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e3fa9d5851e5f8fe38ee9ba228c2b52 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/cbbc695d-d976-4c7e-85fd-1fecf3bbec82_isrForCrosstalkSources_996713_16.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/cbbc695d-d976-4c7e-85fd-1fecf3bbec82_isrForCrosstalkSources_996713_16.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6698b16d8beaf0c7f517ae97faf702ae with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/31e88c38-25f1-4d36-bd08-bee26e8f06fb_isrForCrosstalkSources_996719_2.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/31e88c38-25f1-4d36-bd08-bee26e8f06fb_isrForCrosstalkSources_996719_2.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a847f6157da2a7be29bb974f71f3d9e4 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/7c270093-84f0-48de-aca9-fcfb0e7e4187_isrForCrosstalkSources_996709_41.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/7c270093-84f0-48de-aca9-fcfb0e7e4187_isrForCrosstalkSources_996709_41.stderr 2024-03-15 12:17:57 proc_lsst.multi:170 [INFO] [multi] job_id 17071216 proc_lsst.multi INFO: [multi] job_id 17071216 2024-03-15 12:17:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 12:17:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071216 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071216 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17071216 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-15 12:17:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-15 12:17:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 12:17:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 838a671d0546ca7c9b7ba44a9e963c13 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/d6a5d52f-c503-4d4a-8e14-25c6c4722612_isrForCrosstalkSources_996721_28.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/d6a5d52f-c503-4d4a-8e14-25c6c4722612_isrForCrosstalkSources_996721_28.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f4fe188eed1202d663035e42ae982e0 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/373ee6fe-b81b-4038-8774-f506ee83bed2_isrForCrosstalkSources_996721_38.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/373ee6fe-b81b-4038-8774-f506ee83bed2_isrForCrosstalkSources_996721_38.stderr 2024-03-15 12:17:58 proc_lsst.multi:170 [INFO] [multi] job_id 17071217 proc_lsst.multi INFO: [multi] job_id 17071217 2024-03-15 12:17:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 12:17:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071217 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071217 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17071217 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-15 12:17:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-15 12:17:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 12:17:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c1c9939ca83db69a42ed5f3ae4b8b58 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/c2529fcf-0776-4147-acf2-58d1bcb08633_isrForCrosstalkSources_996711_12.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/c2529fcf-0776-4147-acf2-58d1bcb08633_isrForCrosstalkSources_996711_12.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:17:58 proc_lsst.multi:170 [INFO] [multi] job_id 17071218 proc_lsst.multi INFO: [multi] job_id 17071218 2024-03-15 12:17:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 12:17:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071218 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071218 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17071218 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-15 12:17:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-15 12:17:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 12:17:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec474d3433ea0b3a27b32aa761c3ecd5 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c71cb0b7-d690-4c8d-8408-5686cab8bc1d_isrForCrosstalkSources_996713_53.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c71cb0b7-d690-4c8d-8408-5686cab8bc1d_isrForCrosstalkSources_996713_53.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76f2d7ec7a7c9bca5b47cda6f4057144 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/28704f90-2440-444c-b90e-decd0c78fe29_isrForCrosstalkSources_996714_29.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/28704f90-2440-444c-b90e-decd0c78fe29_isrForCrosstalkSources_996714_29.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0943984d8b74351a177463c3f7f64115 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/fd594b90-b290-4eaa-aa36-09606739e326_isrForCrosstalkSources_996716_61.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/fd594b90-b290-4eaa-aa36-09606739e326_isrForCrosstalkSources_996716_61.stderr 2024-03-15 12:17:59 proc_lsst.multi:170 [INFO] [multi] job_id 17071219 proc_lsst.multi INFO: [multi] job_id 17071219 2024-03-15 12:17:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 12:17:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071219 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071219 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17071219 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-15 12:17:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-15 12:17:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 12:17:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa91cdbda05c45cb904a5deb2bce8d74 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e84f9300-ebd7-46e6-8b15-1bedf6e9993b_isrForCrosstalkSources_996717_38.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e84f9300-ebd7-46e6-8b15-1bedf6e9993b_isrForCrosstalkSources_996717_38.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 966665c8d44e0ae70480637b133bea2f with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/010fde18-fa2c-4a66-acbe-357ff32a6584_isrForCrosstalkSources_996715_54.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/010fde18-fa2c-4a66-acbe-357ff32a6584_isrForCrosstalkSources_996715_54.stderr 2024-03-15 12:17:59 proc_lsst.multi:170 [INFO] [multi] job_id 17071220 proc_lsst.multi INFO: [multi] job_id 17071220 2024-03-15 12:17:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 12:17:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071220 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071220 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17071220 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-15 12:17:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-15 12:17:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:17:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:17:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:17:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:17:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:17:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:17:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:17:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:17:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 12:17:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:17:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a48ce2443deae86cc6a18779a419383 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/42b0ed02-8bca-4918-b8f8-b8b3d11127d7_isrForCrosstalkSources_996708_60.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/42b0ed02-8bca-4918-b8f8-b8b3d11127d7_isrForCrosstalkSources_996708_60.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67515f4a1249057f4e79782fcce145e5 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/500aaaf2-3973-419a-aa06-796590ae93bc_isrForCrosstalkSources_996707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/500aaaf2-3973-419a-aa06-796590ae93bc_isrForCrosstalkSources_996707_62.stderr 2024-03-15 12:18:00 proc_lsst.multi:170 [INFO] [multi] job_id 17071221 proc_lsst.multi INFO: [multi] job_id 17071221 2024-03-15 12:18:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 12:18:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:00 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071221 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071221 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17071221 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-15 12:18:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-15 12:18:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 12:18:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:00 proc_lsst.multi:170 [INFO] [multi] job_id 17071222 proc_lsst.multi INFO: [multi] job_id 17071222 2024-03-15 12:18:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 12:18:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:00 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071222 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071222 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17071222 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-15 12:18:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-15 12:18:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 12:18:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:02 proc_lsst.multi:170 [INFO] [multi] job_id 17071224 proc_lsst.multi INFO: [multi] job_id 17071224 2024-03-15 12:18:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 12:18:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071224 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071224 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17071224 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-15 12:18:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-15 12:18:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 12:18:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ee23799d49d1fdcbf0af232a11d6b17 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/11c8df40-52a3-45f3-989c-57dd70c4f91e_isrForCrosstalkSources_996706_1.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/11c8df40-52a3-45f3-989c-57dd70c4f91e_isrForCrosstalkSources_996706_1.stderr 2024-03-15 12:18:03 proc_lsst.multi:170 [INFO] [multi] job_id 17071225 proc_lsst.multi INFO: [multi] job_id 17071225 2024-03-15 12:18:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 12:18:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071225 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071225 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17071225 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-15 12:18:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-15 12:18:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 12:18:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:03 proc_lsst.multi:170 [INFO] [multi] job_id 17071226 proc_lsst.multi INFO: [multi] job_id 17071226 2024-03-15 12:18:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 12:18:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071226 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071226 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17071226 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-15 12:18:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-15 12:18:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 12:18:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:04 proc_lsst.multi:170 [INFO] [multi] job_id 17071227 proc_lsst.multi INFO: [multi] job_id 17071227 2024-03-15 12:18:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 12:18:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071227 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071227 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17071227 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-15 12:18:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-15 12:18:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 12:18:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:04 proc_lsst.multi:170 [INFO] [multi] job_id 17071228 proc_lsst.multi INFO: [multi] job_id 17071228 2024-03-15 12:18:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 12:18:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071228 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071228 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17071228 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-15 12:18:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-15 12:18:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 12:18:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:05 proc_lsst.multi:170 [INFO] [multi] job_id 17071229 proc_lsst.multi INFO: [multi] job_id 17071229 2024-03-15 12:18:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:05 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17071229 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17071229 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17071229 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-15 12:18:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-15 12:18:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 12:18:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59e9049291e79610ce4bd13155027d63 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/b73a9de8-989c-4fbf-973c-e3e0bfee22ba_isrForCrosstalkSources_996711_23.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/b73a9de8-989c-4fbf-973c-e3e0bfee22ba_isrForCrosstalkSources_996711_23.stderr 2024-03-15 12:18:06 proc_lsst.multi:170 [INFO] [multi] job_id 17071230 proc_lsst.multi INFO: [multi] job_id 17071230 2024-03-15 12:18:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071230 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071230 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17071230 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-15 12:18:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-15 12:18:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:07 proc_lsst.multi:170 [INFO] [multi] job_id 17071231 proc_lsst.multi INFO: [multi] job_id 17071231 2024-03-15 12:18:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 12:18:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071231 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071231 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17071231 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-15 12:18:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-15 12:18:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 12:18:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d92a6b1e597534970680ad3c8768f9f3 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/697bc477-6d7f-41dd-b4b5-67a56945f1e1_isrForCrosstalkSources_996707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/697bc477-6d7f-41dd-b4b5-67a56945f1e1_isrForCrosstalkSources_996707_53.stderr 2024-03-15 12:18:08 proc_lsst.multi:170 [INFO] [multi] job_id 17071232 proc_lsst.multi INFO: [multi] job_id 17071232 2024-03-15 12:18:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 12:18:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071232 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071232 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17071232 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-15 12:18:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-15 12:18:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 12:18:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd0986d450f0c0d7ee2c962edb845266 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/2aaf304a-c941-4892-a23a-98daeedbcc24_isrForCrosstalkSources_996715_25.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/2aaf304a-c941-4892-a23a-98daeedbcc24_isrForCrosstalkSources_996715_25.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f19ea136755e7cb48135ecbeb0e4fac8 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/54affc5d-9f3a-4936-9ddf-b0189a9dd708_isrForCrosstalkSources_996705_37.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/54affc5d-9f3a-4936-9ddf-b0189a9dd708_isrForCrosstalkSources_996705_37.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f553809baa8f2c9a5683982ee13fa1d with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f55f0f18-e923-479f-978e-cc179ac8bba8_isrForCrosstalkSources_996717_17.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f55f0f18-e923-479f-978e-cc179ac8bba8_isrForCrosstalkSources_996717_17.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d669faf718b6758804952ebbb43baef with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/264f0532-8166-41f3-a04d-e519148cb946_isrForCrosstalkSources_996714_40.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/264f0532-8166-41f3-a04d-e519148cb946_isrForCrosstalkSources_996714_40.stderr 2024-03-15 12:18:10 proc_lsst.multi:170 [INFO] [multi] job_id 17071235 proc_lsst.multi INFO: [multi] job_id 17071235 2024-03-15 12:18:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 12:18:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071235 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071235 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17071235 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-15 12:18:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12d4fd325147ea2200f122936ea36975 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/8d1c2462-4c65-4196-8d85-3668b1230408_isrForCrosstalkSources_996708_8.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/8d1c2462-4c65-4196-8d85-3668b1230408_isrForCrosstalkSources_996708_8.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cecc81ad070f2905936dc5840f4268e with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d263253b-9a9d-413b-a2fe-77c503713ec6_isrForCrosstalkSources_996711_36.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d263253b-9a9d-413b-a2fe-77c503713ec6_isrForCrosstalkSources_996711_36.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd9a7081154f2f646b8ace7c42215619 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/ff0e2528-f55d-409f-8421-57a93103e3a6_isrForCrosstalkSources_996711_29.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/ff0e2528-f55d-409f-8421-57a93103e3a6_isrForCrosstalkSources_996711_29.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1a5dcad7fe5f577778e8707c1325e5d with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/eb4801b1-78d7-460a-bc2a-63248d9a9047_isrForCrosstalkSources_996713_6.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/eb4801b1-78d7-460a-bc2a-63248d9a9047_isrForCrosstalkSources_996713_6.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 462557a352d156148525eaf4962b5dbb with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/0f4ca18b-b748-4685-92fd-3c94275a64a1_isrForCrosstalkSources_996709_17.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/0f4ca18b-b748-4685-92fd-3c94275a64a1_isrForCrosstalkSources_996709_17.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e37118ca9556524bb7a9d78cd34589 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0889d857-60ec-4e67-8963-d109af2105a3_isrForCrosstalkSources_996719_47.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0889d857-60ec-4e67-8963-d109af2105a3_isrForCrosstalkSources_996719_47.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e36d4868ce816bfe29ac3ecb6f558a18 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/67121a15-63c4-42b1-bd59-1326b00e8338_isrForCrosstalkSources_996719_30.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/67121a15-63c4-42b1-bd59-1326b00e8338_isrForCrosstalkSources_996719_30.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0d4f34f327fc67c6ced228921f73248 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fecadec6-691f-414c-aa7d-efcfbb49562b_isrForCrosstalkSources_996717_29.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fecadec6-691f-414c-aa7d-efcfbb49562b_isrForCrosstalkSources_996717_29.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8625329a6740b8ab86a2f241b477545b with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/fd6bc526-0331-4eb4-ad42-9cde873b62fa_isrForCrosstalkSources_996718_54.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/fd6bc526-0331-4eb4-ad42-9cde873b62fa_isrForCrosstalkSources_996718_54.stderr 2024-03-15 12:18:10 proc_lsst.multi:170 [INFO] [multi] job_id 17071237 proc_lsst.multi INFO: [multi] job_id 17071237 2024-03-15 12:18:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071237 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071237 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17071237 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-15 12:18:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:11 proc_lsst.multi:170 [INFO] [multi] job_id 17071238 proc_lsst.multi INFO: [multi] job_id 17071238 2024-03-15 12:18:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 12:18:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071238 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071238 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17071238 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-15 12:18:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-15 12:18:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 12:18:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:12 proc_lsst.multi:170 [INFO] [multi] job_id 17071239 proc_lsst.multi INFO: [multi] job_id 17071239 2024-03-15 12:18:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 12:18:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071239 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071239 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17071239 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-15 12:18:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:12 proc_lsst.multi:170 [INFO] [multi] job_id 17071240 proc_lsst.multi INFO: [multi] job_id 17071240 2024-03-15 12:18:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 12:18:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071240 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071240 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17071240 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-15 12:18:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 12:18:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:13 proc_lsst.multi:170 [INFO] [multi] job_id 17071243 proc_lsst.multi INFO: [multi] job_id 17071243 2024-03-15 12:18:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 12:18:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071243 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071243 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17071243 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-15 12:18:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-15 12:18:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 12:18:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:14 proc_lsst.multi:170 [INFO] [multi] job_id 17071244 proc_lsst.multi INFO: [multi] job_id 17071244 2024-03-15 12:18:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 12:18:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071244 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071244 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17071244 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-15 12:18:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:14 proc_lsst.multi:170 [INFO] [multi] job_id 17071245 proc_lsst.multi INFO: [multi] job_id 17071245 2024-03-15 12:18:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 12:18:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071245 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071245 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17071245 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-15 12:18:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:14 proc_lsst.multi:170 [INFO] [multi] job_id 17071247 proc_lsst.multi INFO: [multi] job_id 17071247 2024-03-15 12:18:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 12:18:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071247 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071247 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17071247 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-15 12:18:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 12:18:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:15 proc_lsst.multi:170 [INFO] [multi] job_id 17071248 proc_lsst.multi INFO: [multi] job_id 17071248 2024-03-15 12:18:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 12:18:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071248 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071248 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17071248 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-15 12:18:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da2d4ef067d5643a179bdafbe02d3046 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c7a8a7d3-5233-4a80-8303-f0930f10dcde_isrForCrosstalkSources_996706_18.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c7a8a7d3-5233-4a80-8303-f0930f10dcde_isrForCrosstalkSources_996706_18.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dd48ebc88fe15b0f32c61149c9d02aa with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f44390b7-bfca-4a26-aebe-914a180a3f9d_isrForCrosstalkSources_996707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f44390b7-bfca-4a26-aebe-914a180a3f9d_isrForCrosstalkSources_996707_58.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46ab80783073883fcfe3bab9b44d0b8b with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4221f3b4-fac6-4909-b843-0d9f1c9e40b3_isrForCrosstalkSources_996719_27.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4221f3b4-fac6-4909-b843-0d9f1c9e40b3_isrForCrosstalkSources_996719_27.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efe86b46bc82d9f8013148738f6bf0ae with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/10a4716d-bffe-4b19-9d12-1de0f357603a_isrForCrosstalkSources_996706_34.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/10a4716d-bffe-4b19-9d12-1de0f357603a_isrForCrosstalkSources_996706_34.stderr 2024-03-15 12:18:15 proc_lsst.multi:170 [INFO] [multi] job_id 17071249 proc_lsst.multi INFO: [multi] job_id 17071249 2024-03-15 12:18:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 12:18:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071249 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071249 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17071249 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-15 12:18:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 12:18:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 177993b787293585543f9cbf5ebb0654 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5a91691b-8be4-41d3-b399-d017849df239_isrForCrosstalkSources_996721_34.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5a91691b-8be4-41d3-b399-d017849df239_isrForCrosstalkSources_996721_34.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b84fe8c4ba00f914fe3f1cf4f33ff2cb with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/a76d428d-4bff-409b-8ad4-837ea90cabe6_isrForCrosstalkSources_996710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/a76d428d-4bff-409b-8ad4-837ea90cabe6_isrForCrosstalkSources_996710_23.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bdff9b5546af2605f7badf53b7d215c with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/abb425bd-c715-472d-9b16-64e9e3d2d771_isrForCrosstalkSources_996721_51.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/abb425bd-c715-472d-9b16-64e9e3d2d771_isrForCrosstalkSources_996721_51.stderr 2024-03-15 12:18:16 proc_lsst.multi:170 [INFO] [multi] job_id 17071250 proc_lsst.multi INFO: [multi] job_id 17071250 2024-03-15 12:18:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 12:18:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071250 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071250 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17071250 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-15 12:18:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-15 12:18:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 12:18:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b91dd1837e5231dea0af6715c9bcad2d with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2106dbe2-2503-441c-a6ad-44e72d53b023_isrForCrosstalkSources_996712_14.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2106dbe2-2503-441c-a6ad-44e72d53b023_isrForCrosstalkSources_996712_14.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3e597d79323fe123ba855eb32505911 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/34ed4139-e813-489a-a8c2-1704c12cfc19_isrForCrosstalkSources_996718_35.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/34ed4139-e813-489a-a8c2-1704c12cfc19_isrForCrosstalkSources_996718_35.stderr 2024-03-15 12:18:17 proc_lsst.multi:170 [INFO] [multi] job_id 17071251 proc_lsst.multi INFO: [multi] job_id 17071251 2024-03-15 12:18:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 12:18:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071251 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071251 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17071251 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-15 12:18:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-15 12:18:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 12:18:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbfc833d1510afa34f69aa3575d04e5 with result from task 169 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f9188824-f701-4d69-89f3-bb3c2559bdac_isrForCrosstalkSources_996712_39.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f9188824-f701-4d69-89f3-bb3c2559bdac_isrForCrosstalkSources_996712_39.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8778c57e66e0aaa84fced303639ad713 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/eecab347-4729-49fb-bda5-825e76a358de_isrForCrosstalkSources_996718_36.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/eecab347-4729-49fb-bda5-825e76a358de_isrForCrosstalkSources_996718_36.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aed2dee1f61fc0dd3c8e5e03a11551e5 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/b6bc0f3c-a75e-49b7-9b73-759919150afa_isrForCrosstalkSources_996719_6.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/b6bc0f3c-a75e-49b7-9b73-759919150afa_isrForCrosstalkSources_996719_6.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25abc942ecf0408d24d2cb304ab2e581 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a8c4e0fa-5b92-4ce1-aa6b-fa1f951911c1_isrForCrosstalkSources_996706_29.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a8c4e0fa-5b92-4ce1-aa6b-fa1f951911c1_isrForCrosstalkSources_996706_29.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fe894959d09a4cc388533fdc74ce216 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/63d18a8a-adfc-4005-b737-277a289f60b1_isrForCrosstalkSources_996716_6.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/63d18a8a-adfc-4005-b737-277a289f60b1_isrForCrosstalkSources_996716_6.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffbaf59bae1e13331c812f3fa349c2c1 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/95c4c788-ee8a-4834-8f5e-2c1c80eaef42_isrForCrosstalkSources_996720_7.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/95c4c788-ee8a-4834-8f5e-2c1c80eaef42_isrForCrosstalkSources_996720_7.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd9caf9f858b41e05b95251686d18479 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/8f915c1b-0c12-4504-b223-e4cb552c009b_isrForCrosstalkSources_996709_3.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/8f915c1b-0c12-4504-b223-e4cb552c009b_isrForCrosstalkSources_996709_3.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24af83e4847601241e71c713c5563381 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5b0ab9a5-fde0-45b9-8b64-a68e99768b00_isrForCrosstalkSources_996709_37.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5b0ab9a5-fde0-45b9-8b64-a68e99768b00_isrForCrosstalkSources_996709_37.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1ba8115841858a28a2378e8c976aaa8 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c58ec095-afee-4bfc-85dd-712b731038fc_isrForCrosstalkSources_996709_60.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c58ec095-afee-4bfc-85dd-712b731038fc_isrForCrosstalkSources_996709_60.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a664ca03b5bc4ab2fb00246523ac1ac with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/846beb61-dd43-4b2f-99d1-24825b436769_isrForCrosstalkSources_996715_5.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/846beb61-dd43-4b2f-99d1-24825b436769_isrForCrosstalkSources_996715_5.stderr 2024-03-15 12:18:18 proc_lsst.multi:170 [INFO] [multi] job_id 17071252 proc_lsst.multi INFO: [multi] job_id 17071252 2024-03-15 12:18:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 12:18:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071252 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071252 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17071252 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-15 12:18:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-15 12:18:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 12:18:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 730a946808127e8fdedcd8df93f2b403 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/629f453c-b418-4449-9813-a5c1b387367f_isrForCrosstalkSources_996717_41.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/629f453c-b418-4449-9813-a5c1b387367f_isrForCrosstalkSources_996717_41.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f2a604dbe1df2dfa8877388b73d8d7 with result from task 147 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/87799243-9d84-435c-9f07-cbbec3490e08_isrForCrosstalkSources_996717_53.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/87799243-9d84-435c-9f07-cbbec3490e08_isrForCrosstalkSources_996717_53.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 121e6a537a978e8cffc2c44fbd5ca81a with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e8f17f8e-d006-4044-90dc-a0a9f73e769d_isrForCrosstalkSources_996716_62.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e8f17f8e-d006-4044-90dc-a0a9f73e769d_isrForCrosstalkSources_996716_62.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95a86b372d7361d84f87d033f1f57d9d with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d0f408aa-1721-46e0-a385-1edf379529be_isrForCrosstalkSources_996705_41.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d0f408aa-1721-46e0-a385-1edf379529be_isrForCrosstalkSources_996705_41.stderr 2024-03-15 12:18:19 proc_lsst.multi:170 [INFO] [multi] job_id 17071253 proc_lsst.multi INFO: [multi] job_id 17071253 2024-03-15 12:18:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 12:18:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071253 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071253 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17071253 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-15 12:18:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-15 12:18:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 12:18:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 565bf1ed703474abda63ea7ffcbdecc7 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ad4897ab-f4d9-44fb-99d3-17049694d12f_isrForCrosstalkSources_996717_35.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ad4897ab-f4d9-44fb-99d3-17049694d12f_isrForCrosstalkSources_996717_35.stderr 2024-03-15 12:18:21 proc_lsst.multi:170 [INFO] [multi] job_id 17071257 proc_lsst.multi INFO: [multi] job_id 17071257 2024-03-15 12:18:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 12:18:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071257 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071257 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17071257 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-15 12:18:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-15 12:18:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 12:18:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdfe3fd615f13df145cf885da8192574 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e076adc5-fe61-46e6-97f4-7f6a70a981ca_isrForCrosstalkSources_996717_58.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e076adc5-fe61-46e6-97f4-7f6a70a981ca_isrForCrosstalkSources_996717_58.stderr 2024-03-15 12:18:22 proc_lsst.multi:170 [INFO] [multi] job_id 17071258 proc_lsst.multi INFO: [multi] job_id 17071258 2024-03-15 12:18:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071258 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071258 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17071258 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-15 12:18:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-15 12:18:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a46cc8d96df7e4a51e9159951739034 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/64af2a51-9de0-42db-a795-f718fbd1f248_isrForCrosstalkSources_996715_20.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/64af2a51-9de0-42db-a795-f718fbd1f248_isrForCrosstalkSources_996715_20.stderr 2024-03-15 12:18:23 proc_lsst.multi:170 [INFO] [multi] job_id 17071259 proc_lsst.multi INFO: [multi] job_id 17071259 2024-03-15 12:18:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 12:18:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071259 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071259 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17071259 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-15 12:18:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-15 12:18:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 12:18:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f2752031b62880ae111618a915d05bb with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/37bc1498-fbb8-444f-a674-06f1a2892ead_isrForCrosstalkSources_996719_13.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/37bc1498-fbb8-444f-a674-06f1a2892ead_isrForCrosstalkSources_996719_13.stderr 2024-03-15 12:18:24 proc_lsst.multi:170 [INFO] [multi] job_id 17071261 proc_lsst.multi INFO: [multi] job_id 17071261 2024-03-15 12:18:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 12:18:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071261 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071261 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17071261 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-15 12:18:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-15 12:18:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 12:18:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8c6497e75875425b6b6cfdd983962b3 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/05f59f4c-b6ad-4d39-989d-fc00153ae784_isrForCrosstalkSources_996715_24.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/05f59f4c-b6ad-4d39-989d-fc00153ae784_isrForCrosstalkSources_996715_24.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39704e3cfcd8c57bb8ffd4d0dfa72046 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/16a1baa6-ca9d-452e-897f-56573a9a27de_isrForCrosstalkSources_996707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/16a1baa6-ca9d-452e-897f-56573a9a27de_isrForCrosstalkSources_996707_4.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6314d8b92dadcc4c75cf3ea08320fbec with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f2850d30-bd1a-4a8c-aba1-d2b687e114b2_isrForCrosstalkSources_996708_5.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f2850d30-bd1a-4a8c-aba1-d2b687e114b2_isrForCrosstalkSources_996708_5.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cc378786c9bac146c7d99938ce97b49 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/dbd72520-797e-4b67-a1c0-bf57e17b9daa_isrForCrosstalkSources_996705_15.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/dbd72520-797e-4b67-a1c0-bf57e17b9daa_isrForCrosstalkSources_996705_15.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c21ad54cc03dea1c197fa75bac09ae4b with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/1319832e-e96b-4a80-8a46-b648de45dd13_isrForCrosstalkSources_996710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/1319832e-e96b-4a80-8a46-b648de45dd13_isrForCrosstalkSources_996710_39.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f85d9f210a003657a31634c76b242866 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/0cc3851e-c2b0-4d0f-8cc5-f29db727d8a9_isrForCrosstalkSources_996708_47.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/0cc3851e-c2b0-4d0f-8cc5-f29db727d8a9_isrForCrosstalkSources_996708_47.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210a2b8322802947a77c8fff569037b9 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0c4a3c5c-813d-4f64-9433-4414bffc9df4_isrForCrosstalkSources_996719_59.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0c4a3c5c-813d-4f64-9433-4414bffc9df4_isrForCrosstalkSources_996719_59.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a21af7eddfdb5135a81eb5210277ae0 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/05d6ead2-9a8b-41fc-9b6b-2851d76df13f_isrForCrosstalkSources_996712_12.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/05d6ead2-9a8b-41fc-9b6b-2851d76df13f_isrForCrosstalkSources_996712_12.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f88e6a99bf1f25529b0a3f66559541e5 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5d15fd97-b580-483e-b375-917dae2b2715_isrForCrosstalkSources_996720_49.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5d15fd97-b580-483e-b375-917dae2b2715_isrForCrosstalkSources_996720_49.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c88a0ee80d64e7642825fdbd3ee0c8a1 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/941fb006-20de-4369-9d70-18a7df1db5b3_isrForCrosstalkSources_996721_14.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/941fb006-20de-4369-9d70-18a7df1db5b3_isrForCrosstalkSources_996721_14.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ead0b98dacc06807cf2ee143227698a with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f874b410-6a8b-405e-83c6-b975243075f8_isrForCrosstalkSources_996710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f874b410-6a8b-405e-83c6-b975243075f8_isrForCrosstalkSources_996710_56.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24450d197805fedebdd5170594809027 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/94374850-ef22-46d4-8109-5964eaef9a6e_isrForCrosstalkSources_996713_39.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/94374850-ef22-46d4-8109-5964eaef9a6e_isrForCrosstalkSources_996713_39.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be686c11ceaff443a3547c575759a928 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/28d9893a-c886-4264-b86b-0e0e117abdf3_isrForCrosstalkSources_996710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/28d9893a-c886-4264-b86b-0e0e117abdf3_isrForCrosstalkSources_996710_18.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9103b738167830b722ba1e795553aa1f with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/268acd7b-c651-426a-bd1b-e4f17d5cc5bd_isrForCrosstalkSources_996719_9.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/268acd7b-c651-426a-bd1b-e4f17d5cc5bd_isrForCrosstalkSources_996719_9.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8c241aec0f039d3de27bc120b88da50 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/f56bcb0b-91f1-40a9-b588-fffebd90fb9e_isrForCrosstalkSources_996720_20.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/f56bcb0b-91f1-40a9-b588-fffebd90fb9e_isrForCrosstalkSources_996720_20.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 255f82d689b5ef9930f8435e160fa46b with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c15991ac-f2c2-4125-9422-41e725deb7c3_isrForCrosstalkSources_996708_51.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c15991ac-f2c2-4125-9422-41e725deb7c3_isrForCrosstalkSources_996708_51.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f57c1883a954098c617e6c5cb7c25ef6 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0787b34c-6017-4b8d-b6bf-406915061cc0_isrForCrosstalkSources_996715_4.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0787b34c-6017-4b8d-b6bf-406915061cc0_isrForCrosstalkSources_996715_4.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e567747cb5b72c8278b2b323359625a8 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/64013c5a-265f-4846-8175-06da26538ee1_isrForCrosstalkSources_996715_41.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/64013c5a-265f-4846-8175-06da26538ee1_isrForCrosstalkSources_996715_41.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0ed7674717ebb2d8493cedec5fdb4b2 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ee4c960b-a5b1-42e9-81a4-f4002f2e5193_isrForCrosstalkSources_996707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ee4c960b-a5b1-42e9-81a4-f4002f2e5193_isrForCrosstalkSources_996707_10.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdc5f510c1a0d23f506fc80ed7d05637 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7cdf0e33-7889-42c8-87ae-59eb1b9a4abd_isrForCrosstalkSources_996721_49.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7cdf0e33-7889-42c8-87ae-59eb1b9a4abd_isrForCrosstalkSources_996721_49.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acd74ed55c81ae00bcb7a5ef8d950d71 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/53675ce4-6043-4472-afa6-e91e6250b595_isrForCrosstalkSources_996706_10.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/53675ce4-6043-4472-afa6-e91e6250b595_isrForCrosstalkSources_996706_10.stderr 2024-03-15 12:18:25 proc_lsst.multi:170 [INFO] [multi] job_id 17071263 proc_lsst.multi INFO: [multi] job_id 17071263 2024-03-15 12:18:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 12:18:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071263 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071263 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17071263 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-15 12:18:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-15 12:18:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 12:18:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b753476021afbbda945465290ceba3be with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/a45309e1-0449-4df1-8055-b3728ca4c18a_isrForCrosstalkSources_996715_60.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/a45309e1-0449-4df1-8055-b3728ca4c18a_isrForCrosstalkSources_996715_60.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4bd2f55cbe723c9b4a2f2bf0bacaeb0 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/a8b7ab13-6a36-45bc-aa61-9ca5fd35232b_isrForCrosstalkSources_996707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/a8b7ab13-6a36-45bc-aa61-9ca5fd35232b_isrForCrosstalkSources_996707_55.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e5785cfbd00cdc9f22b108e3e04e217 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/445ffb51-ba77-4b7a-9086-376848caf563_isrForCrosstalkSources_996713_31.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/445ffb51-ba77-4b7a-9086-376848caf563_isrForCrosstalkSources_996713_31.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f21bed746bb62be57ec55613248665b with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d45fa327-4fa6-48ae-a8a5-94ccaa7123a2_isrForCrosstalkSources_996713_58.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d45fa327-4fa6-48ae-a8a5-94ccaa7123a2_isrForCrosstalkSources_996713_58.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 428b01ab7be40737edd71bbdac89ac0b with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/46696e26-72df-4d9b-a7a6-ec4902d78c59_isrForCrosstalkSources_996718_59.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/46696e26-72df-4d9b-a7a6-ec4902d78c59_isrForCrosstalkSources_996718_59.stderr 2024-03-15 12:18:26 proc_lsst.multi:170 [INFO] [multi] job_id 17071264 proc_lsst.multi INFO: [multi] job_id 17071264 2024-03-15 12:18:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 12:18:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071264 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071264 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17071264 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-15 12:18:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-15 12:18:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 12:18:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7b26d6b7155432e15305df4a93cbee4 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/8816fabc-73b6-43c1-ab86-4ddb00aa6b26_isrForCrosstalkSources_996716_42.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/8816fabc-73b6-43c1-ab86-4ddb00aa6b26_isrForCrosstalkSources_996716_42.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 588475a11bbff84ee55fc4e6f1acc480 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/05dab6cf-6307-436d-8e2d-5fde335fde03_isrForCrosstalkSources_996715_22.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/05dab6cf-6307-436d-8e2d-5fde335fde03_isrForCrosstalkSources_996715_22.stderr 2024-03-15 12:18:27 proc_lsst.multi:170 [INFO] [multi] job_id 17071265 proc_lsst.multi INFO: [multi] job_id 17071265 2024-03-15 12:18:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 12:18:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071265 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071265 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17071265 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-15 12:18:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:27 proc_lsst.multi:170 [INFO] [multi] job_id 17071266 proc_lsst.multi INFO: [multi] job_id 17071266 2024-03-15 12:18:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 12:18:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071266 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071266 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17071266 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-15 12:18:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 12:18:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:28 proc_lsst.multi:170 [INFO] [multi] job_id 17071267 proc_lsst.multi INFO: [multi] job_id 17071267 2024-03-15 12:18:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 12:18:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071267 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071267 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17071267 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-15 12:18:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-15 12:18:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 12:18:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4544777f3fd38e8ea42ca8d9860d85bd with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/3407e620-b271-41d1-bd6a-1fd2c1f52abf_isrForCrosstalkSources_996708_52.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/3407e620-b271-41d1-bd6a-1fd2c1f52abf_isrForCrosstalkSources_996708_52.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c7d57259b9beb317b6a006cc4856921 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/ed2d2a3e-8328-45bf-99bb-c973ddb6731f_isrForCrosstalkSources_996721_55.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/ed2d2a3e-8328-45bf-99bb-c973ddb6731f_isrForCrosstalkSources_996721_55.stderr 2024-03-15 12:18:30 proc_lsst.multi:170 [INFO] [multi] job_id 17071268 proc_lsst.multi INFO: [multi] job_id 17071268 2024-03-15 12:18:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 12:18:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071268 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071268 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17071268 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-15 12:18:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:30 proc_lsst.multi:170 [INFO] [multi] job_id 17071269 proc_lsst.multi INFO: [multi] job_id 17071269 2024-03-15 12:18:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 12:18:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071269 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071269 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17071269 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-15 12:18:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 12:18:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:31 proc_lsst.multi:170 [INFO] [multi] job_id 17071270 proc_lsst.multi INFO: [multi] job_id 17071270 2024-03-15 12:18:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071270 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071270 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17071270 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-15 12:18:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:31 proc_lsst.multi:170 [INFO] [multi] job_id 17071271 proc_lsst.multi INFO: [multi] job_id 17071271 2024-03-15 12:18:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-15 12:18:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071271 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071271 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17071271 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-15 12:18:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-15 12:18:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:32 proc_lsst.multi:170 [INFO] [multi] job_id 17071272 proc_lsst.multi INFO: [multi] job_id 17071272 2024-03-15 12:18:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-15 12:18:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071272 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071272 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17071272 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-15 12:18:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:32 proc_lsst.multi:170 [INFO] [multi] job_id 17071273 proc_lsst.multi INFO: [multi] job_id 17071273 2024-03-15 12:18:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-15 12:18:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071273 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071273 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17071273 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-15 12:18:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35c6fab434c2693f45f89316101d8580 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/7a163f6b-a650-43c9-aafa-83c84614ab28_isrForCrosstalkSources_996708_55.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/7a163f6b-a650-43c9-aafa-83c84614ab28_isrForCrosstalkSources_996708_55.stderr 2024-03-15 12:18:32 proc_lsst.multi:170 [INFO] [multi] job_id 17071274 proc_lsst.multi INFO: [multi] job_id 17071274 2024-03-15 12:18:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-15 12:18:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071274 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071274 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17071274 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-15 12:18:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-15 12:18:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbb793d1127296ef6e05db5974ca3ef1 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/b31be672-05a3-4c97-8a2d-535c5cc6ac92_isrForCrosstalkSources_996707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/b31be672-05a3-4c97-8a2d-535c5cc6ac92_isrForCrosstalkSources_996707_19.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7af1d9634c484bf603c48af1d3c2b2b6 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/1f76402c-8200-4f7c-857a-7559ccf36d78_isrForCrosstalkSources_996713_4.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/1f76402c-8200-4f7c-857a-7559ccf36d78_isrForCrosstalkSources_996713_4.stderr 2024-03-15 12:18:33 proc_lsst.multi:170 [INFO] [multi] job_id 17071275 proc_lsst.multi INFO: [multi] job_id 17071275 2024-03-15 12:18:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-15 12:18:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071275 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071275 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17071275 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-15 12:18:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-15 12:18:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-15 12:18:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caee2b1a7e120e03f69f87ea56c12543 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d715ab74-a238-439b-94f2-83a2450eb465_isrForCrosstalkSources_996705_56.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d715ab74-a238-439b-94f2-83a2450eb465_isrForCrosstalkSources_996705_56.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e807e00400010e765d71d02541875cf7 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/20a624d3-9f06-4f3f-91d7-8b786bc9bf63_isrForCrosstalkSources_996711_45.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/20a624d3-9f06-4f3f-91d7-8b786bc9bf63_isrForCrosstalkSources_996711_45.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b63c0052e1011ec4ab7809f2d87b4629 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/210910ce-f1b6-41b2-83c2-36d60a401bd8_isrForCrosstalkSources_996705_6.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/210910ce-f1b6-41b2-83c2-36d60a401bd8_isrForCrosstalkSources_996705_6.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d75b5c3bd4a92df40aa57cc2d106ae7 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1cfb3f4e-311b-4c9d-a292-f5d9cd768494_isrForCrosstalkSources_996705_12.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1cfb3f4e-311b-4c9d-a292-f5d9cd768494_isrForCrosstalkSources_996705_12.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d65679c64ea4cc2f53bcb4e84098bd0 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed9d94dd-be70-4f39-943f-d9288f6cf75a_isrForCrosstalkSources_996709_18.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed9d94dd-be70-4f39-943f-d9288f6cf75a_isrForCrosstalkSources_996709_18.stderr 2024-03-15 12:18:34 proc_lsst.multi:170 [INFO] [multi] job_id 17071276 proc_lsst.multi INFO: [multi] job_id 17071276 2024-03-15 12:18:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-15 12:18:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071276 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071276 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17071276 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-15 12:18:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-15 12:18:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-15 12:18:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0874dcaafd0aca8d98f20279608f8a1d with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/3087a346-5828-469b-87cd-9405038f2ca0_isrForCrosstalkSources_996713_12.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/3087a346-5828-469b-87cd-9405038f2ca0_isrForCrosstalkSources_996713_12.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f342a6f699578f8a3c462dbca63be25b with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/3300418f-9647-4611-b936-4437f99668fd_isrForCrosstalkSources_996717_34.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/3300418f-9647-4611-b936-4437f99668fd_isrForCrosstalkSources_996717_34.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58ffd9d95a4c75cb5b378cc7f228119 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/76eee212-b0a6-4dc9-8bcf-c7cd43711b1c_isrForCrosstalkSources_996707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/76eee212-b0a6-4dc9-8bcf-c7cd43711b1c_isrForCrosstalkSources_996707_1.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b93c0a76f5be884227f0d6815440d631 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e4ef79c5-dbbd-41d3-988b-a80bac02528a_isrForCrosstalkSources_996707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e4ef79c5-dbbd-41d3-988b-a80bac02528a_isrForCrosstalkSources_996707_12.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 505d0f562c1eb9b6d7c0e56c48596c78 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/067341eb-b9f2-4f24-befc-7b10a81fa868_isrForCrosstalkSources_996718_5.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/067341eb-b9f2-4f24-befc-7b10a81fa868_isrForCrosstalkSources_996718_5.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38f7afb66aa70dabfb314d6aebef258a with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8434f56a-f870-4839-8c46-3d2bc9069f80_isrForCrosstalkSources_996718_58.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8434f56a-f870-4839-8c46-3d2bc9069f80_isrForCrosstalkSources_996718_58.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80fdccaf65aa251890b7d3f5a437e238 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/743311d0-23a1-49c7-9a2f-695f5b4c3339_isrForCrosstalkSources_996718_2.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/743311d0-23a1-49c7-9a2f-695f5b4c3339_isrForCrosstalkSources_996718_2.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6dd24cd14a6317192517e840107d623 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/c4272672-9bb5-43c4-b453-d3bad696fb2c_isrForCrosstalkSources_996710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/c4272672-9bb5-43c4-b453-d3bad696fb2c_isrForCrosstalkSources_996710_6.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 657e03b4a349e25e7f0c8e743868b41e with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/86f725d1-4171-42d3-b321-7098ee9ec2c1_isrForCrosstalkSources_996721_1.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/86f725d1-4171-42d3-b321-7098ee9ec2c1_isrForCrosstalkSources_996721_1.stderr 2024-03-15 12:18:35 proc_lsst.multi:170 [INFO] [multi] job_id 17071277 proc_lsst.multi INFO: [multi] job_id 17071277 2024-03-15 12:18:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-15 12:18:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071277 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071277 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17071277 parsl.executors.status_handling INFO: Allocated block ID 93 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93 2024-03-15 12:18:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 2024-03-15 12:18:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-15 12:18:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:36 proc_lsst.multi:170 [INFO] [multi] job_id 17071278 proc_lsst.multi INFO: [multi] job_id 17071278 2024-03-15 12:18:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-15 12:18:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071278 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071278 parsl.executors.status_handling DEBUG: Launched block 93 on executor multi with job ID 17071278 parsl.executors.status_handling INFO: Allocated block ID 94 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-94 2024-03-15 12:18:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5588da037bc82959e17d88795918a808 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/700f984c-957c-4097-be4d-d3bd5330e4ab_isrForCrosstalkSources_996706_30.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/700f984c-957c-4097-be4d-d3bd5330e4ab_isrForCrosstalkSources_996706_30.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 960529964654467b8ead5109db86ec9a with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/16835765-60c2-4d12-a600-f854f3f000c6_isrForCrosstalkSources_996712_43.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/16835765-60c2-4d12-a600-f854f3f000c6_isrForCrosstalkSources_996712_43.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c360c2e347d7f5889ab871a120aeef29 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ea306c0f-d13f-435b-9dbd-ed3575b8a682_isrForCrosstalkSources_996718_19.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ea306c0f-d13f-435b-9dbd-ed3575b8a682_isrForCrosstalkSources_996718_19.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba99e572815653ca1b25c339bb668d67 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4c7e5425-276e-4083-a7fc-cd7fb6c6154c_isrForCrosstalkSources_996714_54.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4c7e5425-276e-4083-a7fc-cd7fb6c6154c_isrForCrosstalkSources_996714_54.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e3a03d5cd175b75e9f4210e9be6e36f with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6fdb446e-6921-457b-a7f2-3e207bab1da8_isrForCrosstalkSources_996709_33.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6fdb446e-6921-457b-a7f2-3e207bab1da8_isrForCrosstalkSources_996709_33.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 283788ff756729c1babd1265f8fb9184 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/4f7896cc-503a-44f1-8f31-cb78fc5619d3_isrForCrosstalkSources_996715_35.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/4f7896cc-503a-44f1-8f31-cb78fc5619d3_isrForCrosstalkSources_996715_35.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 090e1683d69199f8dcbd813932495aba with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/52e40cfe-24a3-4799-8cb6-163c9e2ec483_isrForCrosstalkSources_996716_54.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/52e40cfe-24a3-4799-8cb6-163c9e2ec483_isrForCrosstalkSources_996716_54.stderr 2024-03-15 12:18:36 proc_lsst.multi:170 [INFO] [multi] job_id 17071279 proc_lsst.multi INFO: [multi] job_id 17071279 2024-03-15 12:18:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-15 12:18:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071279 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071279 parsl.executors.status_handling DEBUG: Launched block 94 on executor multi with job ID 17071279 parsl.executors.status_handling INFO: Allocated block ID 95 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-95 2024-03-15 12:18:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-15 12:18:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45f419c258dfe2ee3c063122658f8ae0 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/12ed5003-80aa-4e19-aea3-5e24657030dc_isrForCrosstalkSources_996720_27.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/12ed5003-80aa-4e19-aea3-5e24657030dc_isrForCrosstalkSources_996720_27.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 988c8c049893d48f2928021a1f46bcff with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/7b5793d9-5e54-4b47-8877-8b4d3abc474a_isrForCrosstalkSources_996705_40.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/7b5793d9-5e54-4b47-8877-8b4d3abc474a_isrForCrosstalkSources_996705_40.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f71d0bb77063ee0e7634fe8774bc2af with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/bb79650b-fc1a-4f25-9a36-ea90fa41184d_isrForCrosstalkSources_996715_16.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/bb79650b-fc1a-4f25-9a36-ea90fa41184d_isrForCrosstalkSources_996715_16.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 678d44f2b516d9744e64a96e2ac9c5d1 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3e0c69c5-b8ba-4abc-84fb-6c5bdd7b9bbf_isrForCrosstalkSources_996709_26.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3e0c69c5-b8ba-4abc-84fb-6c5bdd7b9bbf_isrForCrosstalkSources_996709_26.stderr 2024-03-15 12:18:37 proc_lsst.multi:170 [INFO] [multi] job_id 17071280 proc_lsst.multi INFO: [multi] job_id 17071280 2024-03-15 12:18:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-15 12:18:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071280 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071280 parsl.executors.status_handling DEBUG: Launched block 95 on executor multi with job ID 17071280 parsl.executors.status_handling INFO: Allocated block ID 96 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-96 2024-03-15 12:18:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 2024-03-15 12:18:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-15 12:18:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe5b07bccc42553f7ba0d34ddece2528 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/433829a4-d315-4e39-a7e6-a6cc4f277fcb_isrForCrosstalkSources_996706_56.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/433829a4-d315-4e39-a7e6-a6cc4f277fcb_isrForCrosstalkSources_996706_56.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebd8599a71cbaa6c2542e1268642238b with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/d5438126-c046-4533-a272-499256fe27e7_isrForCrosstalkSources_996715_36.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/d5438126-c046-4533-a272-499256fe27e7_isrForCrosstalkSources_996715_36.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b12f2d9cf484460a165218cda3f06c84 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b56d6829-ab0d-492a-9fc7-39995c99383e_isrForCrosstalkSources_996716_21.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b56d6829-ab0d-492a-9fc7-39995c99383e_isrForCrosstalkSources_996716_21.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb8428046a0c8cfca9cbf74ebceb9bed with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f66134a8-b47f-49d1-bad7-440b5e7d5b1a_isrForCrosstalkSources_996712_56.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f66134a8-b47f-49d1-bad7-440b5e7d5b1a_isrForCrosstalkSources_996712_56.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d451cd1246c2085a0c7c6e7f41d0d78c with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8906efb0-a19a-4c8d-bfd5-40c691802d43_isrForCrosstalkSources_996711_21.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8906efb0-a19a-4c8d-bfd5-40c691802d43_isrForCrosstalkSources_996711_21.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3a95f7868137a7b8a91e9f378b46009 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/2970e918-3f5f-4c7a-a6e5-c9480e0085dc_isrForCrosstalkSources_996708_9.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/2970e918-3f5f-4c7a-a6e5-c9480e0085dc_isrForCrosstalkSources_996708_9.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 661f98dddb688fa72b9c84da60ab6d73 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c2641b8d-5897-4799-ab40-3abb06494cc4_isrForCrosstalkSources_996708_48.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c2641b8d-5897-4799-ab40-3abb06494cc4_isrForCrosstalkSources_996708_48.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a38f5091df75aa903295e2fa3af5d0f2 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/edb8cca6-4e2a-4f6a-bc76-5b0c93a30e44_isrForCrosstalkSources_996712_16.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/edb8cca6-4e2a-4f6a-bc76-5b0c93a30e44_isrForCrosstalkSources_996712_16.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ed00693a4fe9ea9f6b0ceff73b86a44 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/011c2a6b-22d3-4e7b-be7b-04ec93c7d5c7_isrForCrosstalkSources_996721_17.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/011c2a6b-22d3-4e7b-be7b-04ec93c7d5c7_isrForCrosstalkSources_996721_17.stderr 2024-03-15 12:18:38 proc_lsst.multi:170 [INFO] [multi] job_id 17071281 proc_lsst.multi INFO: [multi] job_id 17071281 2024-03-15 12:18:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-15 12:18:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071281 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071281 parsl.executors.status_handling DEBUG: Launched block 96 on executor multi with job ID 17071281 parsl.executors.status_handling INFO: Allocated block ID 97 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-97 2024-03-15 12:18:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 2024-03-15 12:18:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-15 12:18:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bb099ff9120689f51acc17ac5083bc0 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5bf7418d-24d5-4e67-b14c-c54f68949e73_isrForCrosstalkSources_996720_44.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5bf7418d-24d5-4e67-b14c-c54f68949e73_isrForCrosstalkSources_996720_44.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa7b781b4812a99f47af136b515f0518 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/4ea9d24a-13e5-424d-b9c2-44b5407dd71f_isrForCrosstalkSources_996709_30.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/4ea9d24a-13e5-424d-b9c2-44b5407dd71f_isrForCrosstalkSources_996709_30.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80a7e4e6d1ab1032b3c2008254b3bea0 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/7285a442-faf5-46d1-9bea-49a8b4abfb0c_isrForCrosstalkSources_996720_53.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/7285a442-faf5-46d1-9bea-49a8b4abfb0c_isrForCrosstalkSources_996720_53.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0f4b442f3b77366d8a22e6ee363dca4 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2e48ec95-dbed-4fe5-88aa-b1e1f3ce2286_isrForCrosstalkSources_996712_6.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2e48ec95-dbed-4fe5-88aa-b1e1f3ce2286_isrForCrosstalkSources_996712_6.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 152e2ccaf3706872669f5ae298c891a6 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e697dd83-0b04-4a48-afa1-78643e22d75d_isrForCrosstalkSources_996708_36.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e697dd83-0b04-4a48-afa1-78643e22d75d_isrForCrosstalkSources_996708_36.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48931b3e5c98b68d3507f0a7b42bd6d7 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9a08f871-1b3d-4e21-be1f-9014e24cf072_isrForCrosstalkSources_996714_41.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9a08f871-1b3d-4e21-be1f-9014e24cf072_isrForCrosstalkSources_996714_41.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc66de61464c705a2026fa2851721c85 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/2cfd1818-a5c3-4fd6-91ca-34c38ef5e312_isrForCrosstalkSources_996706_54.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/2cfd1818-a5c3-4fd6-91ca-34c38ef5e312_isrForCrosstalkSources_996706_54.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc5883c96a2e3192fc9170bafa8383af with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c800ab2d-8207-4523-aa7d-b33d2dacc88e_isrForCrosstalkSources_996712_3.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c800ab2d-8207-4523-aa7d-b33d2dacc88e_isrForCrosstalkSources_996712_3.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42fe58347f77d61f8742f25b6e340ff3 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/c51047bb-d4c9-43a8-81ff-5aab7a923c2e_isrForCrosstalkSources_996711_15.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/c51047bb-d4c9-43a8-81ff-5aab7a923c2e_isrForCrosstalkSources_996711_15.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 205d9d59171007b002ea56ec3a692e08 with result from task 240 2024-03-15 12:18:39 proc_lsst.multi:170 [INFO] [multi] job_id 17071284 proc_lsst.multi INFO: [multi] job_id 17071284 2024-03-15 12:18:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-15 12:18:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071284 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071284 parsl.executors.status_handling DEBUG: Launched block 97 on executor multi with job ID 17071284 parsl.executors.status_handling INFO: Allocated block ID 98 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-98 2024-03-15 12:18:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 2024-03-15 12:18:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-15 12:18:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/f9a03880-bcb7-4641-816c-c7d1ac040f85_isrForCrosstalkSources_996718_47.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/f9a03880-bcb7-4641-816c-c7d1ac040f85_isrForCrosstalkSources_996718_47.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84f47be8f59587955120a32458fb23d3 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/23fa705c-bb80-4fbb-b0db-a7076c6d0bf4_isrForCrosstalkSources_996720_58.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/23fa705c-bb80-4fbb-b0db-a7076c6d0bf4_isrForCrosstalkSources_996720_58.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ba1509d661ee282cb9bad15bbc45b1 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/517e5c25-e45c-4154-8401-3885a20be71d_isrForCrosstalkSources_996719_50.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/517e5c25-e45c-4154-8401-3885a20be71d_isrForCrosstalkSources_996719_50.stderr 2024-03-15 12:18:41 proc_lsst.multi:170 [INFO] [multi] job_id 17071285 proc_lsst.multi INFO: [multi] job_id 17071285 2024-03-15 12:18:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-15 12:18:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071285 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071285 parsl.executors.status_handling DEBUG: Launched block 98 on executor multi with job ID 17071285 parsl.executors.status_handling INFO: Allocated block ID 99 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-99 2024-03-15 12:18:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:41 proc_lsst.multi:170 [INFO] [multi] job_id 17071286 proc_lsst.multi INFO: [multi] job_id 17071286 2024-03-15 12:18:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-15 12:18:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071286 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071286 parsl.executors.status_handling DEBUG: Launched block 99 on executor multi with job ID 17071286 parsl.executors.status_handling INFO: Allocated block ID 100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-100 2024-03-15 12:18:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-15 12:18:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01ebaf4cc43295423aeb97da97ad4db9 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/6eeefdad-17f3-4ff6-be4b-381a1d1bea1c_isrForCrosstalkSources_996705_19.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/6eeefdad-17f3-4ff6-be4b-381a1d1bea1c_isrForCrosstalkSources_996705_19.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8291cf0af0415f51924f13fd362aa64d with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/67b23847-fdf3-4650-be70-844ab0a2eefb_isrForCrosstalkSources_996718_39.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/67b23847-fdf3-4650-be70-844ab0a2eefb_isrForCrosstalkSources_996718_39.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44c4af1618b461ca2b6f387c784fe918 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/f6ddc6a2-15d9-40e1-b971-5af9d724827d_isrForCrosstalkSources_996718_16.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/f6ddc6a2-15d9-40e1-b971-5af9d724827d_isrForCrosstalkSources_996718_16.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb0af4f51d6decc708a2c82a0655dd9c with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/42bd9fd2-ab15-45be-bdb9-ce2c9858f3d9_isrForCrosstalkSources_996707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/42bd9fd2-ab15-45be-bdb9-ce2c9858f3d9_isrForCrosstalkSources_996707_20.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47a994a163449ba31dae19e87ed47ab4 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/cf148278-d0c6-4349-843d-147dca00b76a_isrForCrosstalkSources_996711_49.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/cf148278-d0c6-4349-843d-147dca00b76a_isrForCrosstalkSources_996711_49.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57c5c06c80ee5deb47a41da6da234f1a with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/df580fd4-f578-422c-a420-36c59b5d1800_isrForCrosstalkSources_996716_10.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/df580fd4-f578-422c-a420-36c59b5d1800_isrForCrosstalkSources_996716_10.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36806be52cb8729de8f1b05c6f156165 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c7ca80fb-40b5-4e47-ac9e-f6ed32a23f5b_isrForCrosstalkSources_996706_27.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c7ca80fb-40b5-4e47-ac9e-f6ed32a23f5b_isrForCrosstalkSources_996706_27.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf84963f2892e51039a0db262601e754 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6723bb06-aca1-4f52-8cd5-b3c338f0ff1e_isrForCrosstalkSources_996710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6723bb06-aca1-4f52-8cd5-b3c338f0ff1e_isrForCrosstalkSources_996710_34.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ee5baea854da183dee10a361f243fab with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/d0669d97-c4a9-45b3-a615-0142202b03fe_isrForCrosstalkSources_996706_38.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/d0669d97-c4a9-45b3-a615-0142202b03fe_isrForCrosstalkSources_996706_38.stderr 2024-03-15 12:18:42 proc_lsst.multi:170 [INFO] [multi] job_id 17071288 proc_lsst.multi INFO: [multi] job_id 17071288 2024-03-15 12:18:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-15 12:18:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071288 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071288 parsl.executors.status_handling DEBUG: Launched block 100 on executor multi with job ID 17071288 parsl.executors.status_handling INFO: Allocated block ID 101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-101 2024-03-15 12:18:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 2024-03-15 12:18:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-15 12:18:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1665c51cca6fc04398d184915c756aa with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4570afea-4f9b-4123-a20a-d0a44f7580a0_isrForCrosstalkSources_996707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4570afea-4f9b-4123-a20a-d0a44f7580a0_isrForCrosstalkSources_996707_54.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4cc4e208649cd1a5f2fbddb0d3d3889 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ec54d3cf-e705-4645-bb6f-f23762027732_isrForCrosstalkSources_996705_5.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ec54d3cf-e705-4645-bb6f-f23762027732_isrForCrosstalkSources_996705_5.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43585b4c149c27fc7dc1ae531bd5fca with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/e53c10f1-0698-48a8-81e1-6b0eba9ff462_isrForCrosstalkSources_996712_49.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/e53c10f1-0698-48a8-81e1-6b0eba9ff462_isrForCrosstalkSources_996712_49.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 144f9812e87f1e6845f46b60998c2232 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d2da78b4-60f6-412b-8e86-905663da200b_isrForCrosstalkSources_996709_38.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d2da78b4-60f6-412b-8e86-905663da200b_isrForCrosstalkSources_996709_38.stderr 2024-03-15 12:18:43 proc_lsst.multi:170 [INFO] [multi] job_id 17071289 proc_lsst.multi INFO: [multi] job_id 17071289 2024-03-15 12:18:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-15 12:18:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071289 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071289 parsl.executors.status_handling DEBUG: Launched block 101 on executor multi with job ID 17071289 parsl.executors.status_handling INFO: Allocated block ID 102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-102 2024-03-15 12:18:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 2024-03-15 12:18:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-15 12:18:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:44 proc_lsst.multi:170 [INFO] [multi] job_id 17071290 proc_lsst.multi INFO: [multi] job_id 17071290 2024-03-15 12:18:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-15 12:18:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071290 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071290 parsl.executors.status_handling DEBUG: Launched block 102 on executor multi with job ID 17071290 parsl.executors.status_handling INFO: Allocated block ID 103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-103 2024-03-15 12:18:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 2024-03-15 12:18:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-15 12:18:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fc250c52774dc9b5be794335b787ce0 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8bf09523-e2d3-4b64-9436-a0086ee6064c_isrForCrosstalkSources_996718_43.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8bf09523-e2d3-4b64-9436-a0086ee6064c_isrForCrosstalkSources_996718_43.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b0769d277b4717b70a674669a67d300 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1a74531c-cf5b-445e-8d33-6206c9417e6e_isrForCrosstalkSources_996717_51.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1a74531c-cf5b-445e-8d33-6206c9417e6e_isrForCrosstalkSources_996717_51.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7c2c5f52b4e6b12ccfc181c23716fae with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a94cd407-7dfc-4c5a-9594-76260b3bf673_isrForCrosstalkSources_996705_57.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a94cd407-7dfc-4c5a-9594-76260b3bf673_isrForCrosstalkSources_996705_57.stderr 2024-03-15 12:18:45 proc_lsst.multi:170 [INFO] [multi] job_id 17071291 proc_lsst.multi INFO: [multi] job_id 17071291 2024-03-15 12:18:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-15 12:18:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071291 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071291 parsl.executors.status_handling DEBUG: Launched block 103 on executor multi with job ID 17071291 parsl.executors.status_handling INFO: Allocated block ID 104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-104 2024-03-15 12:18:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:45 proc_lsst.multi:170 [INFO] [multi] job_id 17071292 proc_lsst.multi INFO: [multi] job_id 17071292 2024-03-15 12:18:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-15 12:18:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071292 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071292 parsl.executors.status_handling DEBUG: Launched block 104 on executor multi with job ID 17071292 parsl.executors.status_handling INFO: Allocated block ID 105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-105 2024-03-15 12:18:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-15 12:18:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:46 proc_lsst.multi:170 [INFO] [multi] job_id 17071293 proc_lsst.multi INFO: [multi] job_id 17071293 2024-03-15 12:18:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-15 12:18:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071293 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071293 parsl.executors.status_handling DEBUG: Launched block 105 on executor multi with job ID 17071293 parsl.executors.status_handling INFO: Allocated block ID 106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-106 2024-03-15 12:18:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:46 proc_lsst.multi:170 [INFO] [multi] job_id 17071294 proc_lsst.multi INFO: [multi] job_id 17071294 2024-03-15 12:18:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-15 12:18:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071294 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071294 parsl.executors.status_handling DEBUG: Launched block 106 on executor multi with job ID 17071294 parsl.executors.status_handling INFO: Allocated block ID 107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-107 2024-03-15 12:18:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-15 12:18:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:47 proc_lsst.multi:170 [INFO] [multi] job_id 17071295 proc_lsst.multi INFO: [multi] job_id 17071295 2024-03-15 12:18:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-15 12:18:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071295 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071295 parsl.executors.status_handling DEBUG: Launched block 107 on executor multi with job ID 17071295 parsl.executors.status_handling INFO: Allocated block ID 108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-108 2024-03-15 12:18:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 344dd211469d3ebd69bfa420781bec01 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/65dac767-3034-43c6-9129-374a3f169c3a_isrForCrosstalkSources_996715_26.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/65dac767-3034-43c6-9129-374a3f169c3a_isrForCrosstalkSources_996715_26.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52bbe484e84d714a5e5fcc75087093f6 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/9f0bd5a6-c36a-4709-9665-2f62348fb18e_isrForCrosstalkSources_996713_43.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/9f0bd5a6-c36a-4709-9665-2f62348fb18e_isrForCrosstalkSources_996713_43.stderr 2024-03-15 12:18:47 proc_lsst.multi:170 [INFO] [multi] job_id 17071296 proc_lsst.multi INFO: [multi] job_id 17071296 2024-03-15 12:18:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-15 12:18:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071296 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071296 parsl.executors.status_handling DEBUG: Launched block 108 on executor multi with job ID 17071296 parsl.executors.status_handling INFO: Allocated block ID 109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-109 2024-03-15 12:18:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:47 proc_lsst.multi:170 [INFO] [multi] job_id 17071297 proc_lsst.multi INFO: [multi] job_id 17071297 2024-03-15 12:18:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-15 12:18:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071297 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071297 parsl.executors.status_handling DEBUG: Launched block 109 on executor multi with job ID 17071297 parsl.executors.status_handling INFO: Allocated block ID 110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-110 2024-03-15 12:18:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-15 12:18:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:49 proc_lsst.multi:170 [INFO] [multi] job_id 17071298 proc_lsst.multi INFO: [multi] job_id 17071298 2024-03-15 12:18:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-15 12:18:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071298 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071298 parsl.executors.status_handling DEBUG: Launched block 110 on executor multi with job ID 17071298 parsl.executors.status_handling INFO: Allocated block ID 111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-111 2024-03-15 12:18:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 2024-03-15 12:18:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-15 12:18:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60c43ffc527ae961a803d522daf02524 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/40fbba30-db8a-40ad-bb25-d657c0c21d5b_isrForCrosstalkSources_996706_16.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/40fbba30-db8a-40ad-bb25-d657c0c21d5b_isrForCrosstalkSources_996706_16.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0004b1aa5a0ec4e7423c66545dbef3e5 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/348f4eff-3e0a-4b8b-9a5f-64a3471c9ed7_isrForCrosstalkSources_996705_32.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/348f4eff-3e0a-4b8b-9a5f-64a3471c9ed7_isrForCrosstalkSources_996705_32.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42c51915755e59a090260c78bb182570 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c944ae69-7dc1-4afd-9f89-1f51029b59f0_isrForCrosstalkSources_996714_27.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c944ae69-7dc1-4afd-9f89-1f51029b59f0_isrForCrosstalkSources_996714_27.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad9eadbe032430bef52b64a933537a0e with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/50c449c0-f854-4d42-869b-1d42f3d4f91a_isrForCrosstalkSources_996708_22.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/50c449c0-f854-4d42-869b-1d42f3d4f91a_isrForCrosstalkSources_996708_22.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2085db765fc11c1012750ddc350994b with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/76eb361d-fec9-43df-b266-2bce1fd1ac31_isrForCrosstalkSources_996721_23.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/76eb361d-fec9-43df-b266-2bce1fd1ac31_isrForCrosstalkSources_996721_23.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1abc080be91211f1ea22f702999f601f with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/e081d6b0-c102-47b4-9bf3-45be8aeced66_isrForCrosstalkSources_996719_16.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/e081d6b0-c102-47b4-9bf3-45be8aeced66_isrForCrosstalkSources_996719_16.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2b101330ae430f4a24ae38c5a2fbb44 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cad12935-7e23-4918-8951-9119944c8215_isrForCrosstalkSources_996714_45.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cad12935-7e23-4918-8951-9119944c8215_isrForCrosstalkSources_996714_45.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dfca0d76a73aa80b27277fe988295fc with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f6da887e-5bf4-4eb7-87e4-ca7da68e0d13_isrForCrosstalkSources_996709_21.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f6da887e-5bf4-4eb7-87e4-ca7da68e0d13_isrForCrosstalkSources_996709_21.stderr 2024-03-15 12:18:50 proc_lsst.multi:170 [INFO] [multi] job_id 17071299 proc_lsst.multi INFO: [multi] job_id 17071299 2024-03-15 12:18:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-15 12:18:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071299 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071299 parsl.executors.status_handling DEBUG: Launched block 111 on executor multi with job ID 17071299 parsl.executors.status_handling INFO: Allocated block ID 112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-112 2024-03-15 12:18:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 2024-03-15 12:18:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-15 12:18:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b010ad272d0491c2c3d2f61d6625ac39 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/88e66b53-2865-4d6a-85ed-3facd1e2c7dc_isrForCrosstalkSources_996721_27.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/88e66b53-2865-4d6a-85ed-3facd1e2c7dc_isrForCrosstalkSources_996721_27.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05e7e68db306990aca2a2e7f318fbe54 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/2dec188b-f4eb-49f8-8819-7e449cf88b3a_isrForCrosstalkSources_996717_14.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/2dec188b-f4eb-49f8-8819-7e449cf88b3a_isrForCrosstalkSources_996717_14.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e52896689e3d0275c231a64072c7857 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7b22c0c1-5756-44cd-a60c-ad17bb2b2c34_isrForCrosstalkSources_996714_18.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7b22c0c1-5756-44cd-a60c-ad17bb2b2c34_isrForCrosstalkSources_996714_18.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71c007f10af319993e44b6ae984926fd with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7152f469-7d2b-4d47-9443-b7bd729800bc_isrForCrosstalkSources_996714_7.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7152f469-7d2b-4d47-9443-b7bd729800bc_isrForCrosstalkSources_996714_7.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941487934e0014c67e5fb9f7484eb1af with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/87fba166-8af9-4827-a97b-e588a3df78fd_isrForCrosstalkSources_996712_21.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/87fba166-8af9-4827-a97b-e588a3df78fd_isrForCrosstalkSources_996712_21.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 701ff33c3adb4313a319a2b242ccb8a2 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/fe2a3f56-b9e4-46ea-ab62-ac0ab98663b8_isrForCrosstalkSources_996708_32.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/fe2a3f56-b9e4-46ea-ab62-ac0ab98663b8_isrForCrosstalkSources_996708_32.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ac3c2b8b14af7b690bd012be3a859ce with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e048fe99-c717-4875-a9a2-1f84099b7677_isrForCrosstalkSources_996721_21.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e048fe99-c717-4875-a9a2-1f84099b7677_isrForCrosstalkSources_996721_21.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 053ea9222e5455955cb3d0567422cdcc with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6af3515f-2b66-445a-9c22-98ef8503687c_isrForCrosstalkSources_996709_34.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6af3515f-2b66-445a-9c22-98ef8503687c_isrForCrosstalkSources_996709_34.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a606fbbc82dfdf973abf3b487d5ecde with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f505f6d8-d7fe-46be-a28f-021e55c4b523_isrForCrosstalkSources_996716_23.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f505f6d8-d7fe-46be-a28f-021e55c4b523_isrForCrosstalkSources_996716_23.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cf5813890e8948b0114f22a2d017b52 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7c1235ed-3d1e-42bd-9a8c-33be8fcd6d29_isrForCrosstalkSources_996714_24.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/7c1235ed-3d1e-42bd-9a8c-33be8fcd6d29_isrForCrosstalkSources_996714_24.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c8d790f0ac8a4d06310fc7b8c3903f4 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/80638d52-9011-485e-b53b-c06c2ab97052_isrForCrosstalkSources_996716_18.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/80638d52-9011-485e-b53b-c06c2ab97052_isrForCrosstalkSources_996716_18.stderr 2024-03-15 12:18:51 proc_lsst.multi:170 [INFO] [multi] job_id 17071300 proc_lsst.multi INFO: [multi] job_id 17071300 2024-03-15 12:18:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-15 12:18:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071300 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071300 parsl.executors.status_handling DEBUG: Launched block 112 on executor multi with job ID 17071300 parsl.executors.status_handling INFO: Allocated block ID 113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-113 2024-03-15 12:18:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 2024-03-15 12:18:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-15 12:18:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eb271928c4b1b174d80d5feb53bdef3 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1d6a1b2b-7376-4d0d-bb2e-cd49b7d5aac3_isrForCrosstalkSources_996719_11.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1d6a1b2b-7376-4d0d-bb2e-cd49b7d5aac3_isrForCrosstalkSources_996719_11.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6f73022399eb8b2ce9277d8488ad8ee with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/883b79ed-e1b9-4f21-b0ad-f80f1a3cf165_isrForCrosstalkSources_996709_15.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/883b79ed-e1b9-4f21-b0ad-f80f1a3cf165_isrForCrosstalkSources_996709_15.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c829690290e1f4610f2516915ebc58a with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/3cc834cc-bc21-4796-a710-f7a927231dc9_isrForCrosstalkSources_996715_46.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/3cc834cc-bc21-4796-a710-f7a927231dc9_isrForCrosstalkSources_996715_46.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a3c7ae31e4431b205a0157ee0dfb6b4 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/9f11ad20-386c-43dc-9f4d-2b1606d7bbe3_isrForCrosstalkSources_996713_51.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/9f11ad20-386c-43dc-9f4d-2b1606d7bbe3_isrForCrosstalkSources_996713_51.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f2f8884bace4e75607dbad024169dc7 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a0f4c43e-e37f-4993-b03c-56bae108fdbf_isrForCrosstalkSources_996720_18.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a0f4c43e-e37f-4993-b03c-56bae108fdbf_isrForCrosstalkSources_996720_18.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dce3cd8a82a690c5d9784cdf1571bbc6 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6ff784ea-fd2a-4af5-9d22-2b296a616ac8_isrForCrosstalkSources_996711_22.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6ff784ea-fd2a-4af5-9d22-2b296a616ac8_isrForCrosstalkSources_996711_22.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e3c846834385ed9ed64df2555e277ee with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/267de4a7-61d3-47c3-9f62-13b68ae81503_isrForCrosstalkSources_996713_45.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/267de4a7-61d3-47c3-9f62-13b68ae81503_isrForCrosstalkSources_996713_45.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7b37bff5cdfdba30d593d13cb7184da with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/cd4dcac9-afcf-423b-964c-11466eade8b3_isrForCrosstalkSources_996719_17.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/cd4dcac9-afcf-423b-964c-11466eade8b3_isrForCrosstalkSources_996719_17.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96aef32ba9af69a144a0fa76f3c9d311 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e582e5dc-9552-4dd2-8d9f-b58836194c0d_isrForCrosstalkSources_996708_1.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e582e5dc-9552-4dd2-8d9f-b58836194c0d_isrForCrosstalkSources_996708_1.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d10a54ab3af1483ffe59122e66db08 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/618f55db-81f8-438f-9590-0d00d90e8ea8_isrForCrosstalkSources_996708_44.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/618f55db-81f8-438f-9590-0d00d90e8ea8_isrForCrosstalkSources_996708_44.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8011101a101c8987a1db7573f9775c7 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/41c73bb0-705a-4102-ba7d-2bead7b61316_isrForCrosstalkSources_996716_9.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/41c73bb0-705a-4102-ba7d-2bead7b61316_isrForCrosstalkSources_996716_9.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b55d19c35e00e2e7d3f1a5c6cef8464 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/8be162bb-3e48-497b-99d8-ced93a431739_isrForCrosstalkSources_996708_61.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/8be162bb-3e48-497b-99d8-ced93a431739_isrForCrosstalkSources_996708_61.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 068cfd947c24bf0965e0f74ecc674a5f with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9d8c7646-955c-4fe7-b443-98f0abe85b2f_isrForCrosstalkSources_996708_53.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9d8c7646-955c-4fe7-b443-98f0abe85b2f_isrForCrosstalkSources_996708_53.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e60def3befe8b74a632af3ce465c64a with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5e239d20-3bec-431c-a7ba-2bb9890ed98a_isrForCrosstalkSources_996707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5e239d20-3bec-431c-a7ba-2bb9890ed98a_isrForCrosstalkSources_996707_18.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e681053793d9996e97c61a227137a794 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e25ea7fa-1678-42a9-a5f2-798bf146c4d8_isrForCrosstalkSources_996720_41.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e25ea7fa-1678-42a9-a5f2-798bf146c4d8_isrForCrosstalkSources_996720_41.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74017942e45eb168dc480f15a026184f with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8948b12f-af9a-4a03-a387-7e3c3e726af6_isrForCrosstalkSources_996717_44.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8948b12f-af9a-4a03-a387-7e3c3e726af6_isrForCrosstalkSources_996717_44.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b231eb0d0d0cba93684d49e846fcbaf with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/48ca5fa6-6e3e-46c4-9b92-fa22ca47dd01_isrForCrosstalkSources_996717_42.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/48ca5fa6-6e3e-46c4-9b92-fa22ca47dd01_isrForCrosstalkSources_996717_42.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0478c8fe44409971cbfd84abcefa73d9 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/396254cd-00ae-44f3-8e88-73d3d9df9a70_isrForCrosstalkSources_996711_57.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/396254cd-00ae-44f3-8e88-73d3d9df9a70_isrForCrosstalkSources_996711_57.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f602185c1f3e1dfb3d094193800a5c25 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/7b96c66a-8cd4-4766-9248-5de8648eafbb_isrForCrosstalkSources_996706_55.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/7b96c66a-8cd4-4766-9248-5de8648eafbb_isrForCrosstalkSources_996706_55.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 126c444d7df84bd872336bd8b56b3c43 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/cffca9eb-01ab-4481-8f04-b6ffce2ed6e8_isrForCrosstalkSources_996706_31.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/cffca9eb-01ab-4481-8f04-b6ffce2ed6e8_isrForCrosstalkSources_996706_31.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e210bca4abcc6ffc765241a8bb80dd3e with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ebe97a4c-a7de-486e-9f9c-9afec502f7d0_isrForCrosstalkSources_996718_4.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ebe97a4c-a7de-486e-9f9c-9afec502f7d0_isrForCrosstalkSources_996718_4.stderr 2024-03-15 12:18:53 proc_lsst.multi:170 [INFO] [multi] job_id 17071301 proc_lsst.multi INFO: [multi] job_id 17071301 2024-03-15 12:18:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-15 12:18:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071301 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071301 parsl.executors.status_handling DEBUG: Launched block 113 on executor multi with job ID 17071301 parsl.executors.status_handling INFO: Allocated block ID 114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-114 2024-03-15 12:18:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 2024-03-15 12:18:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-15 12:18:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1b7ed9b1ddc9b2239f093f53df5310c with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8c5b1e4e-dfaa-4d41-8d15-049b08503fe2_isrForCrosstalkSources_996715_30.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8c5b1e4e-dfaa-4d41-8d15-049b08503fe2_isrForCrosstalkSources_996715_30.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae6f8345baadd69c50099b34d123eb07 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3aca22c1-f95d-492b-9b3f-18885944c9ea_isrForCrosstalkSources_996711_44.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3aca22c1-f95d-492b-9b3f-18885944c9ea_isrForCrosstalkSources_996711_44.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ac0389e2a7c299fd7184b2fa377b16d with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8cc06a4e-6171-499f-85c8-5c8cc9e1ed72_isrForCrosstalkSources_996721_48.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8cc06a4e-6171-499f-85c8-5c8cc9e1ed72_isrForCrosstalkSources_996721_48.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8c1512e7f77f96ab0b3a8d8a072f390 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e98873ca-bfb4-4f53-b5ce-2188a2890f23_isrForCrosstalkSources_996708_40.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e98873ca-bfb4-4f53-b5ce-2188a2890f23_isrForCrosstalkSources_996708_40.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5edd8eaee0f55de6297ee3c0e9510375 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b2f66708-2b66-4886-aa31-470ccf10ed9e_isrForCrosstalkSources_996712_25.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b2f66708-2b66-4886-aa31-470ccf10ed9e_isrForCrosstalkSources_996712_25.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe82ae5272f106fecb2a9c4b635b7699 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/1c353220-69a8-4525-9d84-5f1572cbf0bd_isrForCrosstalkSources_996713_19.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/1c353220-69a8-4525-9d84-5f1572cbf0bd_isrForCrosstalkSources_996713_19.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1404d8d0797477560349ed2e44bee5f1 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/8d2dfd76-7e8c-49bc-86a9-ca38573533ee_isrForCrosstalkSources_996714_5.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/8d2dfd76-7e8c-49bc-86a9-ca38573533ee_isrForCrosstalkSources_996714_5.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d64f514294b1e92f48172cb489af2fc1 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ceab6fb6-eb19-4833-a421-25889f8678bc_isrForCrosstalkSources_996705_52.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ceab6fb6-eb19-4833-a421-25889f8678bc_isrForCrosstalkSources_996705_52.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e543570d117febc3ed2c0efc06dd62bf with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9564b2b3-9fb4-4565-96eb-64720e4d373a_isrForCrosstalkSources_996718_53.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9564b2b3-9fb4-4565-96eb-64720e4d373a_isrForCrosstalkSources_996718_53.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68775510d82d2256db746bfa8aa7ba66 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/9d552b8c-b44d-4aa8-b85a-ebc71acb5da5_isrForCrosstalkSources_996709_53.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/9d552b8c-b44d-4aa8-b85a-ebc71acb5da5_isrForCrosstalkSources_996709_53.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85b9232330c43d793dd227cc4596ed0 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/8c89a177-5adf-43f2-8a82-c7523082db3c_isrForCrosstalkSources_996705_61.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/8c89a177-5adf-43f2-8a82-c7523082db3c_isrForCrosstalkSources_996705_61.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d74485165d04238b55cc6af5be1d020 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/091e59fd-7151-4fe6-9fef-5bc86a968bd2_isrForCrosstalkSources_996712_52.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/091e59fd-7151-4fe6-9fef-5bc86a968bd2_isrForCrosstalkSources_996712_52.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb689fa1b75a9f349e475315fc77142d with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/678a033b-1396-492b-8115-28d43b7db879_isrForCrosstalkSources_996718_41.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/678a033b-1396-492b-8115-28d43b7db879_isrForCrosstalkSources_996718_41.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 867889f818eda809645155ebfbce1e84 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/320823f3-1bb7-4777-943f-aa0258e8d7b8_isrForCrosstalkSources_996713_59.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/320823f3-1bb7-4777-943f-aa0258e8d7b8_isrForCrosstalkSources_996713_59.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc0541be4a32b1200d60bdb5aad7e50a with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6ff6882f-e8ca-413a-9ef6-7c7cf8d4e98c_isrForCrosstalkSources_996710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6ff6882f-e8ca-413a-9ef6-7c7cf8d4e98c_isrForCrosstalkSources_996710_5.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf6a8fc0da6ad2e598d1f8d2fa74f3f3 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/1a846c31-affe-4909-ba41-5c3e8887a999_isrForCrosstalkSources_996710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/1a846c31-affe-4909-ba41-5c3e8887a999_isrForCrosstalkSources_996710_50.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd2c709d99997b11037923faed726fdc with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/6913f9f4-b476-4e37-8c72-cbe61f71919d_isrForCrosstalkSources_996717_15.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/6913f9f4-b476-4e37-8c72-cbe61f71919d_isrForCrosstalkSources_996717_15.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1516622b485a1a06229d964c6813735f with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5a5f249c-c6fc-4c3c-ad21-a2bac68cf970_isrForCrosstalkSources_996719_15.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5a5f249c-c6fc-4c3c-ad21-a2bac68cf970_isrForCrosstalkSources_996719_15.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84c6500d58b8f3ef78496efbbcb2e9e0 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b0417ab8-adde-4320-9f95-e51521f5f0c4_isrForCrosstalkSources_996705_49.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b0417ab8-adde-4320-9f95-e51521f5f0c4_isrForCrosstalkSources_996705_49.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a3f620aa76f6185071d55568a8fe6a4 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2aeb3d99-bedf-4230-ad36-2e93052891ea_isrForCrosstalkSources_996714_25.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2aeb3d99-bedf-4230-ad36-2e93052891ea_isrForCrosstalkSources_996714_25.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec98e1a4944df5e035961701a09d8d64 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3e000eb2-b1f5-4e69-a3f7-0167ffc48013_isrForCrosstalkSources_996705_9.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3e000eb2-b1f5-4e69-a3f7-0167ffc48013_isrForCrosstalkSources_996705_9.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c8c46d5a31ef207dbd25f840eca3b1a with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fee7b79a-5844-4321-858f-fa7109531a21_isrForCrosstalkSources_996705_27.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fee7b79a-5844-4321-858f-fa7109531a21_isrForCrosstalkSources_996705_27.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0919f29433f0781fb4fe7b65838c46cc with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ad9632a2-5a65-4607-bd7d-237e5cb9b198_isrForCrosstalkSources_996720_47.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ad9632a2-5a65-4607-bd7d-237e5cb9b198_isrForCrosstalkSources_996720_47.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c7a9f87014345ae1449a1f53cce98bc with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/07cec972-c502-442b-a670-87d0a52ffa13_isrForCrosstalkSources_996716_34.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/07cec972-c502-442b-a670-87d0a52ffa13_isrForCrosstalkSources_996716_34.stderr 2024-03-15 12:18:56 proc_lsst.multi:170 [INFO] [multi] job_id 17071302 proc_lsst.multi INFO: [multi] job_id 17071302 2024-03-15 12:18:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-15 12:18:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071302 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071302 parsl.executors.status_handling DEBUG: Launched block 114 on executor multi with job ID 17071302 parsl.executors.status_handling INFO: Allocated block ID 115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-115 2024-03-15 12:18:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 2024-03-15 12:18:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-15 12:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf3ff9071a7a158648e0e08478370b16 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/14b2cbe8-ce03-4d1c-89a9-2dd990c72a92_isrForCrosstalkSources_996716_19.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/14b2cbe8-ce03-4d1c-89a9-2dd990c72a92_isrForCrosstalkSources_996716_19.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 757baf3e3093bda7b64db5f53f94a5fd with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fe0f1f71-fec8-46e8-b26e-260c7420b3f0_isrForCrosstalkSources_996705_62.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fe0f1f71-fec8-46e8-b26e-260c7420b3f0_isrForCrosstalkSources_996705_62.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84945da44d2185d502fcc4f638136e69 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0d4f7a63-dc6b-43ce-a27a-06739b3594fb_isrForCrosstalkSources_996720_30.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0d4f7a63-dc6b-43ce-a27a-06739b3594fb_isrForCrosstalkSources_996720_30.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b9925144b2a1f4dc64bdb8b040ea0b7 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/96116b52-4360-46a0-b302-8fecaa20ae4a_isrForCrosstalkSources_996709_44.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/96116b52-4360-46a0-b302-8fecaa20ae4a_isrForCrosstalkSources_996709_44.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb919412763834527b6bd5ad9de3dfdf with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a3a13918-16ed-4f8a-8cdd-f6b5c917a6f5_isrForCrosstalkSources_996709_52.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a3a13918-16ed-4f8a-8cdd-f6b5c917a6f5_isrForCrosstalkSources_996709_52.stderr 2024-03-15 12:18:57 proc_lsst.multi:170 [INFO] [multi] job_id 17071303 proc_lsst.multi INFO: [multi] job_id 17071303 2024-03-15 12:18:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-15 12:18:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071303 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071303 parsl.executors.status_handling DEBUG: Launched block 115 on executor multi with job ID 17071303 parsl.executors.status_handling INFO: Allocated block ID 116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-116 2024-03-15 12:18:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63e01d7e0be8c56be45ca2d81f196479 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/d33a9b46-3b1f-4f34-8c74-10c0f048b19c_isrForCrosstalkSources_996714_62.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/d33a9b46-3b1f-4f34-8c74-10c0f048b19c_isrForCrosstalkSources_996714_62.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d9700bafc0dad65d27143d00a9574cc with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/bdbad49e-4eb0-4349-9fc1-d844245c78fc_isrForCrosstalkSources_996719_60.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/bdbad49e-4eb0-4349-9fc1-d844245c78fc_isrForCrosstalkSources_996719_60.stderr 2024-03-15 12:18:57 proc_lsst.multi:170 [INFO] [multi] job_id 17071304 proc_lsst.multi INFO: [multi] job_id 17071304 2024-03-15 12:18:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-15 12:18:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071304 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071304 parsl.executors.status_handling DEBUG: Launched block 116 on executor multi with job ID 17071304 parsl.executors.status_handling INFO: Allocated block ID 117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-117 2024-03-15 12:18:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-15 12:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:58 proc_lsst.multi:170 [INFO] [multi] job_id 17071305 proc_lsst.multi INFO: [multi] job_id 17071305 2024-03-15 12:18:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-15 12:18:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071305 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071305 parsl.executors.status_handling DEBUG: Launched block 117 on executor multi with job ID 17071305 parsl.executors.status_handling INFO: Allocated block ID 118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-118 2024-03-15 12:18:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 2024-03-15 12:18:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-15 12:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:18:59 proc_lsst.multi:170 [INFO] [multi] job_id 17071306 proc_lsst.multi INFO: [multi] job_id 17071306 2024-03-15 12:18:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-15 12:18:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071306 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071306 parsl.executors.status_handling DEBUG: Launched block 118 on executor multi with job ID 17071306 parsl.executors.status_handling INFO: Allocated block ID 119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-119 2024-03-15 12:18:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 2024-03-15 12:18:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:18:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:18:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:18:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:18:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-15 12:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:18:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:00 proc_lsst.multi:170 [INFO] [multi] job_id 17071307 proc_lsst.multi INFO: [multi] job_id 17071307 2024-03-15 12:19:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-15 12:19:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071307 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071307 parsl.executors.status_handling DEBUG: Launched block 119 on executor multi with job ID 17071307 parsl.executors.status_handling INFO: Allocated block ID 120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-120 2024-03-15 12:19:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4a8dc2ecbb86ade8826a873f704898a with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8f05ad04-f7d0-4f34-990e-c204aa2939cd_isrForCrosstalkSources_996718_10.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8f05ad04-f7d0-4f34-990e-c204aa2939cd_isrForCrosstalkSources_996718_10.stderr parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1414670f38583e75f44d910d47ed38fd with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ffc9149f-90e0-4856-b332-ddf301fe109f_isrForCrosstalkSources_996706_49.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ffc9149f-90e0-4856-b332-ddf301fe109f_isrForCrosstalkSources_996706_49.stderr 2024-03-15 12:19:00 proc_lsst.multi:170 [INFO] [multi] job_id 17071308 proc_lsst.multi INFO: [multi] job_id 17071308 2024-03-15 12:19:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-15 12:19:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071308 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071308 parsl.executors.status_handling DEBUG: Launched block 120 on executor multi with job ID 17071308 parsl.executors.status_handling INFO: Allocated block ID 121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-121 2024-03-15 12:19:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-15 12:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67683ca6856bf5c2416f1ecdf2b9dc3d with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8766a960-c236-420e-a99d-db7acf344b07_isrForCrosstalkSources_996712_2.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8766a960-c236-420e-a99d-db7acf344b07_isrForCrosstalkSources_996712_2.stderr 2024-03-15 12:19:01 proc_lsst.multi:170 [INFO] [multi] job_id 17071309 proc_lsst.multi INFO: [multi] job_id 17071309 2024-03-15 12:19:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-15 12:19:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071309 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071309 parsl.executors.status_handling DEBUG: Launched block 121 on executor multi with job ID 17071309 parsl.executors.status_handling INFO: Allocated block ID 122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-122 2024-03-15 12:19:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 040400a9b54866526581836e45527a24 with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4b7a88b1-702e-4b4b-8a8e-f79b5453d945_isrForCrosstalkSources_996719_20.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4b7a88b1-702e-4b4b-8a8e-f79b5453d945_isrForCrosstalkSources_996719_20.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25688f3b298b3c9541b2309ff808d05 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/e831800d-f695-422c-9939-ccbdde843214_isrForCrosstalkSources_996714_21.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/e831800d-f695-422c-9939-ccbdde843214_isrForCrosstalkSources_996714_21.stderr 2024-03-15 12:19:01 proc_lsst.multi:170 [INFO] [multi] job_id 17071310 proc_lsst.multi INFO: [multi] job_id 17071310 2024-03-15 12:19:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-15 12:19:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071310 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071310 parsl.executors.status_handling DEBUG: Launched block 122 on executor multi with job ID 17071310 parsl.executors.status_handling INFO: Allocated block ID 123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-123 2024-03-15 12:19:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-15 12:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e39830cc05fef4e6623346c7ec3aeba2 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed4af404-c071-4797-ac42-c36634cd5ec0_isrForCrosstalkSources_996709_46.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ed4af404-c071-4797-ac42-c36634cd5ec0_isrForCrosstalkSources_996709_46.stderr 2024-03-15 12:19:02 proc_lsst.multi:170 [INFO] [multi] job_id 17071311 proc_lsst.multi INFO: [multi] job_id 17071311 2024-03-15 12:19:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-15 12:19:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071311 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071311 parsl.executors.status_handling DEBUG: Launched block 123 on executor multi with job ID 17071311 parsl.executors.status_handling INFO: Allocated block ID 124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-124 2024-03-15 12:19:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8d4e68afd9139a130dea46b63f1c913 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3e89ebbf-7f22-49d4-921f-68529b4ef50c_isrForCrosstalkSources_996707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3e89ebbf-7f22-49d4-921f-68529b4ef50c_isrForCrosstalkSources_996707_33.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 618db001481a0771fbed6bd0ca7ed814 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d317cfad-afc8-48ec-bec9-c567cbe78c19_isrForCrosstalkSources_996705_55.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/d317cfad-afc8-48ec-bec9-c567cbe78c19_isrForCrosstalkSources_996705_55.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a09d14c30f1d3df2d182eee78c2898e with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2259e3c9-386b-471d-b47a-13fd79835aa1_isrForCrosstalkSources_996710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2259e3c9-386b-471d-b47a-13fd79835aa1_isrForCrosstalkSources_996710_41.stderr 2024-03-15 12:19:02 proc_lsst.multi:170 [INFO] [multi] job_id 17071312 proc_lsst.multi INFO: [multi] job_id 17071312 2024-03-15 12:19:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 69 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-15 12:19:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071312 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071312 parsl.executors.status_handling DEBUG: Launched block 124 on executor multi with job ID 17071312 parsl.executors.status_handling INFO: Allocated block ID 125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-125 2024-03-15 12:19:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 69 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f768389d9e77c40784a04101d159f8a with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/674017d1-a186-4f34-b4d3-3c45345241e3_isrForCrosstalkSources_996716_41.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/674017d1-a186-4f34-b4d3-3c45345241e3_isrForCrosstalkSources_996716_41.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b99d6d3c6f8cae80a9c84965c8aa3443 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0d4eae95-cf15-4853-b57e-d22ca644410c_isrForCrosstalkSources_996715_42.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0d4eae95-cf15-4853-b57e-d22ca644410c_isrForCrosstalkSources_996715_42.stderr 2024-03-15 12:19:02 proc_lsst.multi:170 [INFO] [multi] job_id 17071313 proc_lsst.multi INFO: [multi] job_id 17071313 2024-03-15 12:19:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-15 12:19:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071313 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071313 parsl.executors.status_handling DEBUG: Launched block 125 on executor multi with job ID 17071313 parsl.executors.status_handling INFO: Allocated block ID 126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-126 2024-03-15 12:19:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-15 12:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 370cdb2e1a6c994347a7901a9d1cb458 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/cc47aa71-b156-448a-84d8-76ff2d242f41_isrForCrosstalkSources_996721_37.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/cc47aa71-b156-448a-84d8-76ff2d242f41_isrForCrosstalkSources_996721_37.stderr 2024-03-15 12:19:03 proc_lsst.multi:170 [INFO] [multi] job_id 17071314 proc_lsst.multi INFO: [multi] job_id 17071314 2024-03-15 12:19:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-15 12:19:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071314 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071314 parsl.executors.status_handling DEBUG: Launched block 126 on executor multi with job ID 17071314 parsl.executors.status_handling INFO: Allocated block ID 127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-127 2024-03-15 12:19:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d93501997846f4bad4efcaebc5ec718 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/598c3750-7dc6-4ec5-963d-13f42cc21db3_isrForCrosstalkSources_996710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/598c3750-7dc6-4ec5-963d-13f42cc21db3_isrForCrosstalkSources_996710_57.stderr parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9463395332383404389f2b45b634151d with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/707f6d4a-4284-48af-b4c4-a1dcee552e49_isrForCrosstalkSources_996711_56.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/707f6d4a-4284-48af-b4c4-a1dcee552e49_isrForCrosstalkSources_996711_56.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dcb12b6e123d90e651d6f06b4d8fd83 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/722c1973-cae8-45cd-9584-a3e9f346bd54_isrForCrosstalkSources_996711_35.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/722c1973-cae8-45cd-9584-a3e9f346bd54_isrForCrosstalkSources_996711_35.stderr 2024-03-15 12:19:03 proc_lsst.multi:170 [INFO] [multi] job_id 17071315 proc_lsst.multi INFO: [multi] job_id 17071315 2024-03-15 12:19:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-15 12:19:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071315 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071315 parsl.executors.status_handling DEBUG: Launched block 127 on executor multi with job ID 17071315 parsl.executors.status_handling INFO: Allocated block ID 128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-128 2024-03-15 12:19:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-15 12:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a7be501452d07e2f483dba0ecfe7169 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/c7e7f013-abd9-463a-b2ba-824ab3f151ef_isrForCrosstalkSources_996720_3.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/c7e7f013-abd9-463a-b2ba-824ab3f151ef_isrForCrosstalkSources_996720_3.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49443d767720fba26e090ddb6935ba8b with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f971576e-a8c2-4a52-8336-748d58b03916_isrForCrosstalkSources_996705_48.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f971576e-a8c2-4a52-8336-748d58b03916_isrForCrosstalkSources_996705_48.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276265d049905e3d271ec73bd3e2ad8f with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/caef9236-54d9-4861-8917-7cd4834340ba_isrForCrosstalkSources_996719_24.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/caef9236-54d9-4861-8917-7cd4834340ba_isrForCrosstalkSources_996719_24.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8664f6604a16d088806f9169378d4f67 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dc43bfd7-6654-48f3-884e-5fc6d8e67bca_isrForCrosstalkSources_996711_32.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dc43bfd7-6654-48f3-884e-5fc6d8e67bca_isrForCrosstalkSources_996711_32.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47918cc0f18c96959fae94e22f56d580 with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/9533f097-04ce-4c5f-9a96-b13cf3af78d6_isrForCrosstalkSources_996721_11.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/9533f097-04ce-4c5f-9a96-b13cf3af78d6_isrForCrosstalkSources_996721_11.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e29c753e37adc30281974c0da9dbd7c with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/fdb3c65d-22b6-47dd-aaa6-82fb26e092d7_isrForCrosstalkSources_996714_26.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/fdb3c65d-22b6-47dd-aaa6-82fb26e092d7_isrForCrosstalkSources_996714_26.stderr 2024-03-15 12:19:04 proc_lsst.multi:170 [INFO] [multi] job_id 17071316 proc_lsst.multi INFO: [multi] job_id 17071316 2024-03-15 12:19:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-15 12:19:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071316 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071316 parsl.executors.status_handling DEBUG: Launched block 128 on executor multi with job ID 17071316 parsl.executors.status_handling INFO: Allocated block ID 129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-129 2024-03-15 12:19:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 2024-03-15 12:19:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-15 12:19:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abf290f40e63ba4a54834151033d9594 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ddbc1b14-e28b-4c87-80d8-6e443d517d96_isrForCrosstalkSources_996707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ddbc1b14-e28b-4c87-80d8-6e443d517d96_isrForCrosstalkSources_996707_17.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed0e7fb943d3c810b68a00c5c6be28ff with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/5c7d167e-b384-41d3-bb62-312032f5c353_isrForCrosstalkSources_996714_19.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/5c7d167e-b384-41d3-bb62-312032f5c353_isrForCrosstalkSources_996714_19.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db533fdbfdd892febb4bfd7f292131fc with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e7341843-bcd7-4e9b-8f28-10e9dd301cef_isrForCrosstalkSources_996707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e7341843-bcd7-4e9b-8f28-10e9dd301cef_isrForCrosstalkSources_996707_25.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f45e7b7a73c3072f44e8374a8434d4f5 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/550f7ad4-ca6a-4564-ad03-c67b520246cc_isrForCrosstalkSources_996720_16.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/550f7ad4-ca6a-4564-ad03-c67b520246cc_isrForCrosstalkSources_996720_16.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d7b77b77b20ec0ce3aa19334d3506c8 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/1fec5e67-feeb-4b09-bc7c-f49f50bda9b1_isrForCrosstalkSources_996720_6.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/1fec5e67-feeb-4b09-bc7c-f49f50bda9b1_isrForCrosstalkSources_996720_6.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6d2254dc14237632040865ebdc1c8f5 with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f5fb4d68-f423-4aac-a505-efffda43a5f2_isrForCrosstalkSources_996711_3.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f5fb4d68-f423-4aac-a505-efffda43a5f2_isrForCrosstalkSources_996711_3.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2edb4d72c42cf35093d6e222a293b17d with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/6db471f4-6508-475f-92da-7861386d2eff_isrForCrosstalkSources_996706_43.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/6db471f4-6508-475f-92da-7861386d2eff_isrForCrosstalkSources_996706_43.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210119dd5da1fa851261bcb3a4c78a72 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ebff6dbd-c31a-4b7e-97e3-4b7dc9e4a5e6_isrForCrosstalkSources_996718_37.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ebff6dbd-c31a-4b7e-97e3-4b7dc9e4a5e6_isrForCrosstalkSources_996718_37.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fab55d584f6b21c77a69f33f7d1c6d0b with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f9a65cce-a4a7-48ca-be2a-f5e23ca7537f_isrForCrosstalkSources_996705_11.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f9a65cce-a4a7-48ca-be2a-f5e23ca7537f_isrForCrosstalkSources_996705_11.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0af4332ed1ad11497caab29048da146e with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a75325d3-f207-4d5e-94c3-63bc6a1e6ef4_isrForCrosstalkSources_996712_27.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a75325d3-f207-4d5e-94c3-63bc6a1e6ef4_isrForCrosstalkSources_996712_27.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1c3862ca9e77f40eef21d9059fb03f4 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/dc5d68d8-afa7-41f3-b12a-eb4119073240_isrForCrosstalkSources_996718_3.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/dc5d68d8-afa7-41f3-b12a-eb4119073240_isrForCrosstalkSources_996718_3.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d99994a761fd832811a01164b038914f with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2390a6a2-407a-4b6d-988c-c70b16a16ea2_isrForCrosstalkSources_996713_61.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2390a6a2-407a-4b6d-988c-c70b16a16ea2_isrForCrosstalkSources_996713_61.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f6fc612d17a60e21fd644be1a6961d9 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8e220760-6494-4809-b9d5-9eddfd6cb9a8_isrForCrosstalkSources_996712_53.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8e220760-6494-4809-b9d5-9eddfd6cb9a8_isrForCrosstalkSources_996712_53.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0c35251435c12aa49dc503e2d385f58 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/18908587-af0e-4287-8f4d-648c2dd7fdd4_isrForCrosstalkSources_996709_19.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/18908587-af0e-4287-8f4d-648c2dd7fdd4_isrForCrosstalkSources_996709_19.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01e0c441ead883ff7a5ce75e9ace16e6 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/eb5dc226-e578-4975-9ac6-3c52860defd9_isrForCrosstalkSources_996705_45.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/eb5dc226-e578-4975-9ac6-3c52860defd9_isrForCrosstalkSources_996705_45.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de0bb7d0ec9009566fd6d3364a203d2c with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/b601081a-fee9-4865-b4b4-1d294eb8206d_isrForCrosstalkSources_996718_34.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/b601081a-fee9-4865-b4b4-1d294eb8206d_isrForCrosstalkSources_996718_34.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3028f344abf5385d79dbd8f6411aa994 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ff7fb28a-1787-4100-b300-dd3a87ee4399_isrForCrosstalkSources_996717_52.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ff7fb28a-1787-4100-b300-dd3a87ee4399_isrForCrosstalkSources_996717_52.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db078b4b81f7f8cd4dcef6f91242d6c with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f06c94fe-4b10-4c9f-8039-d0c521af4ea8_isrForCrosstalkSources_996709_51.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f06c94fe-4b10-4c9f-8039-d0c521af4ea8_isrForCrosstalkSources_996709_51.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62d8d6336e06d8d6c783fb5a823e93eb with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a546f8d7-df09-4364-9ca9-1ccff58fb09b_isrForCrosstalkSources_996709_13.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a546f8d7-df09-4364-9ca9-1ccff58fb09b_isrForCrosstalkSources_996709_13.stderr parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94219e936e18f98772a79877a9c2fccd with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8ee3a6a1-0371-4ef0-a2a8-43a208a9cc37_isrForCrosstalkSources_996711_47.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8ee3a6a1-0371-4ef0-a2a8-43a208a9cc37_isrForCrosstalkSources_996711_47.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bef247555db1d585ce53d0ad6540bcf with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f4827e38-3e83-44dc-97f9-978e0d1824c3_isrForCrosstalkSources_996713_52.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f4827e38-3e83-44dc-97f9-978e0d1824c3_isrForCrosstalkSources_996713_52.stderr parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a13f620065e944ecaf8ae113e23d1e71 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/da2d356f-cd44-40f2-8f02-6bf84535ac2f_isrForCrosstalkSources_996715_9.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/da2d356f-cd44-40f2-8f02-6bf84535ac2f_isrForCrosstalkSources_996715_9.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 352b40aaedea52c50e34df0d50932d0e with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/b089c0ae-8e0c-4d2e-b8df-ab164abb522a_isrForCrosstalkSources_996719_32.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/b089c0ae-8e0c-4d2e-b8df-ab164abb522a_isrForCrosstalkSources_996719_32.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e9bdadcf31fef087e98f3fa3bd3713c with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8e1af386-5650-4d01-a067-d9a397ee036f_isrForCrosstalkSources_996712_20.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/8e1af386-5650-4d01-a067-d9a397ee036f_isrForCrosstalkSources_996712_20.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23e081fb5b62abe6415ecea05538eac5 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/629fbf0d-114c-495b-a5d4-7524581070f6_isrForCrosstalkSources_996712_15.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/629fbf0d-114c-495b-a5d4-7524581070f6_isrForCrosstalkSources_996712_15.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 694d85479688e0cd00b35374c3e2b3e8 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/cdaf9858-65f1-4e6b-a949-489a57956073_isrForCrosstalkSources_996720_29.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/cdaf9858-65f1-4e6b-a949-489a57956073_isrForCrosstalkSources_996720_29.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9397dc4528b4249daabca798a5137e4a with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/9768af4c-7c9c-4259-8d02-d1eab997d8b7_isrForCrosstalkSources_996715_1.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/9768af4c-7c9c-4259-8d02-d1eab997d8b7_isrForCrosstalkSources_996715_1.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa707a2f6c94bc37f7bebb4ae78db4ce with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c659d2c5-068d-4757-a529-69eaa6252721_isrForCrosstalkSources_996714_2.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c659d2c5-068d-4757-a529-69eaa6252721_isrForCrosstalkSources_996714_2.stderr 2024-03-15 12:19:05 proc_lsst.multi:170 [INFO] [multi] job_id 17071317 proc_lsst.multi INFO: [multi] job_id 17071317 2024-03-15 12:19:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-15 12:19:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071317 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071317 parsl.executors.status_handling DEBUG: Launched block 129 on executor multi with job ID 17071317 parsl.executors.status_handling INFO: Allocated block ID 130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-130 2024-03-15 12:19:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 2024-03-15 12:19:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-15 12:19:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6a59605fb739fd27eebbba5ed5eae03 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/df7a351e-d4d9-45a9-829c-48aeb253063b_isrForCrosstalkSources_996720_21.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/df7a351e-d4d9-45a9-829c-48aeb253063b_isrForCrosstalkSources_996720_21.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc2bdf970d15215c1022ae429c0cf2a6 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/785e5351-8a98-47ce-9e5e-2dbc2fdaad49_isrForCrosstalkSources_996717_5.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/785e5351-8a98-47ce-9e5e-2dbc2fdaad49_isrForCrosstalkSources_996717_5.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a7b292103b354f51df72c428fc55c41 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/cb40ae91-510a-4956-b4bd-056fcd9be8f3_isrForCrosstalkSources_996715_7.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/cb40ae91-510a-4956-b4bd-056fcd9be8f3_isrForCrosstalkSources_996715_7.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9aca2c62f07fa97990451c248a3b79e with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/36bf3096-8153-47c0-8c8e-4a6b47a02bf0_isrForCrosstalkSources_996711_28.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/36bf3096-8153-47c0-8c8e-4a6b47a02bf0_isrForCrosstalkSources_996711_28.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be148b18f197f45633b0ab9eb6c2aba9 with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/b62416e2-384b-4c1c-9fff-410e96d94ab2_isrForCrosstalkSources_996720_12.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/b62416e2-384b-4c1c-9fff-410e96d94ab2_isrForCrosstalkSources_996720_12.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcae5e961db1ce17f9a02c3b6c06a924 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/7d38e4ea-e6fe-4bd3-ab38-a491ea8554fe_isrForCrosstalkSources_996710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/7d38e4ea-e6fe-4bd3-ab38-a491ea8554fe_isrForCrosstalkSources_996710_10.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fb9ed95d70c6155d9a52e582592cb80 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1161420e-515a-4d78-a407-f3218aa43fe0_isrForCrosstalkSources_996719_23.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1161420e-515a-4d78-a407-f3218aa43fe0_isrForCrosstalkSources_996719_23.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ab36a0c6a75df148fe0ef9ad8e34e85 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/fdb2f32f-b8eb-41df-8b7d-317fb81e846f_isrForCrosstalkSources_996720_9.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/fdb2f32f-b8eb-41df-8b7d-317fb81e846f_isrForCrosstalkSources_996720_9.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0c884e3a168a30758d44f2d59040ea5 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6dbc93ac-7406-4607-a263-b95ef2715b99_isrForCrosstalkSources_996710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6dbc93ac-7406-4607-a263-b95ef2715b99_isrForCrosstalkSources_996710_1.stderr 2024-03-15 12:19:06 proc_lsst.multi:170 [INFO] [multi] job_id 17071318 proc_lsst.multi INFO: [multi] job_id 17071318 2024-03-15 12:19:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-15 12:19:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071318 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071318 parsl.executors.status_handling DEBUG: Launched block 130 on executor multi with job ID 17071318 parsl.executors.status_handling INFO: Allocated block ID 131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-131 2024-03-15 12:19:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 2024-03-15 12:19:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-15 12:19:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6353a576426c086220753581643a7a0 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b01532a8-3f9d-455d-ade8-a12ed24798a1_isrForCrosstalkSources_996705_35.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/b01532a8-3f9d-455d-ade8-a12ed24798a1_isrForCrosstalkSources_996705_35.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d98cc3dc5953224dd30188dbb3db4d4 with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/9c388c2c-92ec-4e3c-8219-2c76d35549ba_isrForCrosstalkSources_996709_35.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/9c388c2c-92ec-4e3c-8219-2c76d35549ba_isrForCrosstalkSources_996709_35.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a317b945c7aaa5df92bf9188e5df5b93 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f5a54abe-9e12-4862-9f19-3bdb267ee07a_isrForCrosstalkSources_996710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f5a54abe-9e12-4862-9f19-3bdb267ee07a_isrForCrosstalkSources_996710_43.stderr 2024-03-15 12:19:07 proc_lsst.multi:170 [INFO] [multi] job_id 17071319 proc_lsst.multi INFO: [multi] job_id 17071319 2024-03-15 12:19:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-15 12:19:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071319 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071319 parsl.executors.status_handling DEBUG: Launched block 131 on executor multi with job ID 17071319 parsl.executors.status_handling INFO: Allocated block ID 132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-132 2024-03-15 12:19:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 2024-03-15 12:19:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-15 12:19:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db66909c25535a4f3f3e532d22ebd0eb with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9075fb3f-4c48-4fed-8e54-1eebf8424c0b_isrForCrosstalkSources_996706_15.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9075fb3f-4c48-4fed-8e54-1eebf8424c0b_isrForCrosstalkSources_996706_15.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0d8e22be2a3e61b713691be8c73cf47 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fc625098-375a-4d56-a377-4f5703d880a5_isrForCrosstalkSources_996705_28.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fc625098-375a-4d56-a377-4f5703d880a5_isrForCrosstalkSources_996705_28.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e48021d85ab00667ba48196286dc6b03 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/48f81c04-d93a-41a2-9066-22ed728d0ed4_isrForCrosstalkSources_996719_35.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/48f81c04-d93a-41a2-9066-22ed728d0ed4_isrForCrosstalkSources_996719_35.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c3a8130abd462ba26a55849a0d58837 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1b750868-0d4c-4368-9c34-27084e35289e_isrForCrosstalkSources_996712_50.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1b750868-0d4c-4368-9c34-27084e35289e_isrForCrosstalkSources_996712_50.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be7ebb5ea838234d3e6a9670b5cfaaba with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/ef864896-801e-4e9d-a6ea-f192e0093ed2_isrForCrosstalkSources_996712_22.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/ef864896-801e-4e9d-a6ea-f192e0093ed2_isrForCrosstalkSources_996712_22.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79d011369ec4f5658dfe97b3a9f21cac with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/94ca0ad9-b6a1-424f-8e3f-b7c9ab261134_isrForCrosstalkSources_996712_29.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/94ca0ad9-b6a1-424f-8e3f-b7c9ab261134_isrForCrosstalkSources_996712_29.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e1c3d17d7ca0333667e7c3a4db66f48 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/408d5709-6480-4a33-898c-d3ac59fe0210_isrForCrosstalkSources_996714_31.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/408d5709-6480-4a33-898c-d3ac59fe0210_isrForCrosstalkSources_996714_31.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8672e2f631314d2e22d34492f25a9439 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ecba0ccf-1fb7-4f79-b597-e70d7baba7d6_isrForCrosstalkSources_996714_48.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ecba0ccf-1fb7-4f79-b597-e70d7baba7d6_isrForCrosstalkSources_996714_48.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbbd6a575d502324751d8459ada5f390 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c09f7128-73fe-4d56-bec4-47a9199dc104_isrForCrosstalkSources_996713_49.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c09f7128-73fe-4d56-bec4-47a9199dc104_isrForCrosstalkSources_996713_49.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdb10b42a58842595e3e05d3ca0474f1 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1615b6b2-3c29-46dc-a9b1-64663c46ece5_isrForCrosstalkSources_996712_54.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1615b6b2-3c29-46dc-a9b1-64663c46ece5_isrForCrosstalkSources_996712_54.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c117c829aea4aea9491d0aed5827323e with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e7b8af98-10c1-4086-8d0b-907604259740_isrForCrosstalkSources_996715_49.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e7b8af98-10c1-4086-8d0b-907604259740_isrForCrosstalkSources_996715_49.stderr 2024-03-15 12:19:08 proc_lsst.multi:170 [INFO] [multi] job_id 17071320 proc_lsst.multi INFO: [multi] job_id 17071320 2024-03-15 12:19:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-15 12:19:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071320 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071320 parsl.executors.status_handling DEBUG: Launched block 132 on executor multi with job ID 17071320 parsl.executors.status_handling INFO: Allocated block ID 133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-133 2024-03-15 12:19:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 2024-03-15 12:19:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-15 12:19:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01dd7eb5521fd3c009181f624d6a9c4a with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/cb700fae-c862-4ad1-b86d-4aec3bdf2385_isrForCrosstalkSources_996721_24.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/cb700fae-c862-4ad1-b86d-4aec3bdf2385_isrForCrosstalkSources_996721_24.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d10df93ead263d64783b573c658a55a1 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/28e686c8-5be1-4826-923f-919f79d000e3_isrForCrosstalkSources_996714_20.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/28e686c8-5be1-4826-923f-919f79d000e3_isrForCrosstalkSources_996714_20.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c30be4e46c104c6af08cb8d4f5536327 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/13e74549-61ce-43a6-bf92-0f553a8b1d08_isrForCrosstalkSources_996705_1.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/13e74549-61ce-43a6-bf92-0f553a8b1d08_isrForCrosstalkSources_996705_1.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd9e145b0a781fed8be0081c17491fd with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/02111ca9-4405-4590-9736-1de9c40325bd_isrForCrosstalkSources_996720_22.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/02111ca9-4405-4590-9736-1de9c40325bd_isrForCrosstalkSources_996720_22.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dada57d5312d1ec357a73326bb91eb9 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7fe197b3-3980-4c2e-9cb7-ec1f82b238a5_isrForCrosstalkSources_996721_18.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7fe197b3-3980-4c2e-9cb7-ec1f82b238a5_isrForCrosstalkSources_996721_18.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ddc2a27dfef81d7d8b0518dc8e1b250 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/5fc87a6f-05a6-465b-be05-93cba2ff574b_isrForCrosstalkSources_996712_11.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/5fc87a6f-05a6-465b-be05-93cba2ff574b_isrForCrosstalkSources_996712_11.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79457d4894828fc7446aad47b71f5242 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3df4386d-7fdf-4eaa-b457-f0cc7fc9f4d8_isrForCrosstalkSources_996711_51.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3df4386d-7fdf-4eaa-b457-f0cc7fc9f4d8_isrForCrosstalkSources_996711_51.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d282498bf38e9334f1bfcfe0d0e6557 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/679fd183-8559-4967-aa4f-67123a3c5eec_isrForCrosstalkSources_996720_11.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/679fd183-8559-4967-aa4f-67123a3c5eec_isrForCrosstalkSources_996720_11.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47b49820af5a3f873801c6c7227878e6 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/169d5d68-8956-43d1-98fd-2e3b47615bb3_isrForCrosstalkSources_996713_7.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/169d5d68-8956-43d1-98fd-2e3b47615bb3_isrForCrosstalkSources_996713_7.stderr parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a5baf6456562db259fdfce386161e36 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7d1abd81-55ed-4415-80dc-9144f9c512ff_isrForCrosstalkSources_996716_11.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7d1abd81-55ed-4415-80dc-9144f9c512ff_isrForCrosstalkSources_996716_11.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22e8c64db817a7b1fd415673d8fe5e36 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/aa16f71b-95d5-4505-a844-ff0ee5bb01db_isrForCrosstalkSources_996718_1.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/aa16f71b-95d5-4505-a844-ff0ee5bb01db_isrForCrosstalkSources_996718_1.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 762ad349a84d26c587894402cb09f63a with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/04543c7d-1dda-4eff-961a-b9875d54b224_isrForCrosstalkSources_996714_39.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/04543c7d-1dda-4eff-961a-b9875d54b224_isrForCrosstalkSources_996714_39.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7c87ea5dd5b7158e36b704e4b98ad0c with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e960b2a8-8e42-4eaa-806d-fc91016d04a8_isrForCrosstalkSources_996708_15.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e960b2a8-8e42-4eaa-806d-fc91016d04a8_isrForCrosstalkSources_996708_15.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ad7ae59801bc46805427ed156f3783e with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e628d147-f95f-4c6e-a87c-99a0c7b02e88_isrForCrosstalkSources_996721_42.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e628d147-f95f-4c6e-a87c-99a0c7b02e88_isrForCrosstalkSources_996721_42.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 790c0f65cea819d07808d8fed3e267c4 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/578428ec-f5a8-4fc4-8116-fec6a9fa5bfd_isrForCrosstalkSources_996717_54.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/578428ec-f5a8-4fc4-8116-fec6a9fa5bfd_isrForCrosstalkSources_996717_54.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5257ace4a2fa61ece72b68dca4ae1a9c with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/03b1d77e-0252-452f-ae7d-746ef9be58a0_isrForCrosstalkSources_996707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/03b1d77e-0252-452f-ae7d-746ef9be58a0_isrForCrosstalkSources_996707_6.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb9ce7a0113362ec5f3af8145ecca3c1 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c1167a60-7767-4863-85ab-e3bfc9dfb853_isrForCrosstalkSources_996719_58.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c1167a60-7767-4863-85ab-e3bfc9dfb853_isrForCrosstalkSources_996719_58.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fb4acb062c7ccc2549eda52003835f4 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5069c245-57d4-4f9f-b993-4d3fd0f0e523_isrForCrosstalkSources_996709_28.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5069c245-57d4-4f9f-b993-4d3fd0f0e523_isrForCrosstalkSources_996709_28.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25579499852f85da383aff88413a3df3 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/de3085cb-5a1d-432f-aa0e-25bdcdafb20f_isrForCrosstalkSources_996715_61.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/de3085cb-5a1d-432f-aa0e-25bdcdafb20f_isrForCrosstalkSources_996715_61.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3ba8a7a95a50cdf817f273e27e1b300 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5da6bcb8-fd77-417e-9d5a-89053043879a_isrForCrosstalkSources_996718_55.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5da6bcb8-fd77-417e-9d5a-89053043879a_isrForCrosstalkSources_996718_55.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 106879f14d855567790400ce6bfbe4e8 with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9b904e8e-bafa-416b-8d10-ee0a4669403f_isrForCrosstalkSources_996718_50.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9b904e8e-bafa-416b-8d10-ee0a4669403f_isrForCrosstalkSources_996718_50.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bc1cddfbd795262de2ac6878e4d8154 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/85d73c33-ba28-46a4-9faf-3b7f3e819e74_isrForCrosstalkSources_996711_50.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/85d73c33-ba28-46a4-9faf-3b7f3e819e74_isrForCrosstalkSources_996711_50.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d10574b149cddd6a401c7915dae6a86a with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/5a6b7e98-6f9e-493b-8d76-16a94d481082_isrForCrosstalkSources_996715_39.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/5a6b7e98-6f9e-493b-8d76-16a94d481082_isrForCrosstalkSources_996715_39.stderr 2024-03-15 12:19:09 proc_lsst.multi:170 [INFO] [multi] job_id 17071321 proc_lsst.multi INFO: [multi] job_id 17071321 2024-03-15 12:19:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-15 12:19:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071321 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071321 parsl.executors.status_handling DEBUG: Launched block 133 on executor multi with job ID 17071321 parsl.executors.status_handling INFO: Allocated block ID 134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-134 2024-03-15 12:19:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 2024-03-15 12:19:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-15 12:19:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d49711054369cf273b06dfe5f5d9af7 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/260c1a47-1a4a-4ade-ba17-094f3c376b65_isrForCrosstalkSources_996713_25.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/260c1a47-1a4a-4ade-ba17-094f3c376b65_isrForCrosstalkSources_996713_25.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cc26bc51ece49488c575364852af629 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/81ce262d-8152-4fc1-96b3-f5e97404d43c_isrForCrosstalkSources_996719_22.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/81ce262d-8152-4fc1-96b3-f5e97404d43c_isrForCrosstalkSources_996719_22.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 624ad1e443127cd03d979b696c940175 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3bb25ed6-dd03-438b-80c7-e96bb8e62605_isrForCrosstalkSources_996720_4.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3bb25ed6-dd03-438b-80c7-e96bb8e62605_isrForCrosstalkSources_996720_4.stderr 2024-03-15 12:19:11 proc_lsst.multi:170 [INFO] [multi] job_id 17071322 proc_lsst.multi INFO: [multi] job_id 17071322 2024-03-15 12:19:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-15 12:19:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071322 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071322 parsl.executors.status_handling DEBUG: Launched block 134 on executor multi with job ID 17071322 parsl.executors.status_handling INFO: Allocated block ID 135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-135 2024-03-15 12:19:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99759b40e9a56b15e81678dc7b83e7bc with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/94bc3a17-9523-43ba-9a90-38204e069c68_isrForCrosstalkSources_996717_50.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/94bc3a17-9523-43ba-9a90-38204e069c68_isrForCrosstalkSources_996717_50.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ff1f79f6a43bfb7dd1baead5dc482a6 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/bc0d21f2-d32e-4f33-9a6c-99f7d6f5e89b_isrForCrosstalkSources_996717_11.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/bc0d21f2-d32e-4f33-9a6c-99f7d6f5e89b_isrForCrosstalkSources_996717_11.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cfdc902d63fe0f9d32866b88d3cd83e with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/dbf53692-7f21-4b33-ad44-01339cb8489a_isrForCrosstalkSources_996714_49.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/dbf53692-7f21-4b33-ad44-01339cb8489a_isrForCrosstalkSources_996714_49.stderr 2024-03-15 12:19:11 proc_lsst.multi:170 [INFO] [multi] job_id 17071323 proc_lsst.multi INFO: [multi] job_id 17071323 2024-03-15 12:19:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-15 12:19:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071323 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071323 parsl.executors.status_handling DEBUG: Launched block 135 on executor multi with job ID 17071323 parsl.executors.status_handling INFO: Allocated block ID 136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-136 2024-03-15 12:19:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-15 12:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ee6835c37eeda207a84276baef35c5a with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fcedf03a-febc-47ba-b349-bfcee5180c84_isrForCrosstalkSources_996721_10.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fcedf03a-febc-47ba-b349-bfcee5180c84_isrForCrosstalkSources_996721_10.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be158c9d00eae31baf49325353cf9883 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f7e7f7ac-7d38-4b48-a953-1b364507afcb_isrForCrosstalkSources_996710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/f7e7f7ac-7d38-4b48-a953-1b364507afcb_isrForCrosstalkSources_996710_51.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04f58d77ee83a9b561a8c72d344b4611 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7421d754-ac32-4a0c-836a-8cb587cc01ca_isrForCrosstalkSources_996712_8.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7421d754-ac32-4a0c-836a-8cb587cc01ca_isrForCrosstalkSources_996712_8.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cd4ab6a773e937cfdb7b813bde108bd with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/daa825ae-824f-4149-a0e0-e5a3a487f7cf_isrForCrosstalkSources_996713_24.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/daa825ae-824f-4149-a0e0-e5a3a487f7cf_isrForCrosstalkSources_996713_24.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0ae39ada2a369088e3f893ab55edd86 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c0be4e60-4337-4ced-bfa0-7794dfa972ab_isrForCrosstalkSources_996705_4.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c0be4e60-4337-4ced-bfa0-7794dfa972ab_isrForCrosstalkSources_996705_4.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6506772ba629890c2a35c8adb3d9360a with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/56c91f8f-ebaa-49bc-8c69-e00be677245c_isrForCrosstalkSources_996708_46.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/56c91f8f-ebaa-49bc-8c69-e00be677245c_isrForCrosstalkSources_996708_46.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42f42bbc80905eb7cbd8e75fd54746eb with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f4028fe0-4683-4e10-92e7-ef45c9ca0195_isrForCrosstalkSources_996705_17.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f4028fe0-4683-4e10-92e7-ef45c9ca0195_isrForCrosstalkSources_996705_17.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 761577e7887d68f5416422ee7095d7ed with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/76543138-7d27-48a7-a22d-36a6887ad526_isrForCrosstalkSources_996707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/76543138-7d27-48a7-a22d-36a6887ad526_isrForCrosstalkSources_996707_46.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 271743c167fbd2c0f7fa72f89782181b with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/d6997dcd-6da5-419f-b34c-c88d3e9f9d4f_isrForCrosstalkSources_996717_26.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/d6997dcd-6da5-419f-b34c-c88d3e9f9d4f_isrForCrosstalkSources_996717_26.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 892970fd10e3a1f63fc8a7e03a23d626 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/503bd415-06d3-4cec-9b8e-bb68c2a05b04_isrForCrosstalkSources_996711_8.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/503bd415-06d3-4cec-9b8e-bb68c2a05b04_isrForCrosstalkSources_996711_8.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5df5272eb1db9648bb92b12a67cea8d with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/e2d08e00-5def-42b3-afc9-3ebb36d8de28_isrForCrosstalkSources_996710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/e2d08e00-5def-42b3-afc9-3ebb36d8de28_isrForCrosstalkSources_996710_25.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db66f03fff4ef06a91fbaecf7aa506e8 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/550e1b07-2234-4a74-9699-4c1a8b0a7ebf_isrForCrosstalkSources_996707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/550e1b07-2234-4a74-9699-4c1a8b0a7ebf_isrForCrosstalkSources_996707_35.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba6391dda86bf543b2ad9ced7ed49f05 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/4e2772f4-2d6c-4ef3-9e7f-8cadc2918c55_isrForCrosstalkSources_996720_26.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/4e2772f4-2d6c-4ef3-9e7f-8cadc2918c55_isrForCrosstalkSources_996720_26.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6af601bedeaf748e4210243cbc9d2441 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6a94448a-affb-4d35-ad77-f329522388ac_isrForCrosstalkSources_996709_29.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/6a94448a-affb-4d35-ad77-f329522388ac_isrForCrosstalkSources_996709_29.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 421c1ba728dea8c2a388e1bb85f34da9 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/42c197e8-25dc-4e16-bcd6-f353ac73ab39_isrForCrosstalkSources_996719_56.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/42c197e8-25dc-4e16-bcd6-f353ac73ab39_isrForCrosstalkSources_996719_56.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f62d9854ee7f0cf6b90dd818dc05cbd4 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/34acdd12-1c01-4b73-9a0e-4855d4d19060_isrForCrosstalkSources_996719_29.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/34acdd12-1c01-4b73-9a0e-4855d4d19060_isrForCrosstalkSources_996719_29.stderr parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daa5b3f4c6b939e21be04de4062ea950 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3da5e83c-bbce-4e9d-995e-5af3f49bf702_isrForCrosstalkSources_996719_34.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3da5e83c-bbce-4e9d-995e-5af3f49bf702_isrForCrosstalkSources_996719_34.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79b7417aa78a46c4c5f945fc8fd7f74e with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/d5470117-627b-4a5c-9e34-4c20435668f2_isrForCrosstalkSources_996718_33.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/d5470117-627b-4a5c-9e34-4c20435668f2_isrForCrosstalkSources_996718_33.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43f277ec17805c0341857ad0b24242fe with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ec28e420-e14e-40dc-88ce-b824014214f4_isrForCrosstalkSources_996714_37.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ec28e420-e14e-40dc-88ce-b824014214f4_isrForCrosstalkSources_996714_37.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3115fc4031b47e67708ed167b53c399 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/48ca19f9-b86d-4a0c-895e-2778a84d6a42_isrForCrosstalkSources_996711_1.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/48ca19f9-b86d-4a0c-895e-2778a84d6a42_isrForCrosstalkSources_996711_1.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cbbdd1260e67dfc48cc06bf43429cbd with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/538c608c-2637-410d-9789-071b12d78a2b_isrForCrosstalkSources_996705_44.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/538c608c-2637-410d-9789-071b12d78a2b_isrForCrosstalkSources_996705_44.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d611b32cf0d3f001c023f43353f96dd with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/1f3a0057-7a3d-4c0f-ac4e-58f6dcc41a25_isrForCrosstalkSources_996711_39.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/1f3a0057-7a3d-4c0f-ac4e-58f6dcc41a25_isrForCrosstalkSources_996711_39.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4729893a0119ffc358c6d39f017c2e4 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/bb22b37d-6e3c-44d4-8604-341f2c595bde_isrForCrosstalkSources_996708_23.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/bb22b37d-6e3c-44d4-8604-341f2c595bde_isrForCrosstalkSources_996708_23.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d52c53b689f82d06592c0588fde3d2bb with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b1e46745-33c4-443f-90df-02564ee571fc_isrForCrosstalkSources_996716_40.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b1e46745-33c4-443f-90df-02564ee571fc_isrForCrosstalkSources_996716_40.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db9fb0c3d33244a61f51bb75e9deb657 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/f2f42c21-1c2d-4113-a60f-23fbbc5721e5_isrForCrosstalkSources_996715_3.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/f2f42c21-1c2d-4113-a60f-23fbbc5721e5_isrForCrosstalkSources_996715_3.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6cca541aee49ab14d49fa9b197d1325 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/769531ae-9e3b-496e-a9a0-d1ecb0db871e_isrForCrosstalkSources_996717_28.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/769531ae-9e3b-496e-a9a0-d1ecb0db871e_isrForCrosstalkSources_996717_28.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c494d2301888db250785477bda74543 with result from task 341 2024-03-15 12:19:12 proc_lsst.multi:170 [INFO] [multi] job_id 17071324 proc_lsst.multi INFO: [multi] job_id 17071324 2024-03-15 12:19:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-15 12:19:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071324 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071324 parsl.executors.status_handling DEBUG: Launched block 136 on executor multi with job ID 17071324 parsl.executors.status_handling INFO: Allocated block ID 137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-137 2024-03-15 12:19:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 2024-03-15 12:19:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-15 12:19:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ce20cc28-5b54-4ef9-81e7-0b3e32b5dffc_isrForCrosstalkSources_996717_40.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/ce20cc28-5b54-4ef9-81e7-0b3e32b5dffc_isrForCrosstalkSources_996717_40.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 507699e3241954244b35a03d23dfc3ed with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/0818ade6-a9d8-47ab-8503-63ef498e8576_isrForCrosstalkSources_996718_45.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/0818ade6-a9d8-47ab-8503-63ef498e8576_isrForCrosstalkSources_996718_45.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc97c3db5be9cc0c875570a22de0ac6f with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/67d38442-8c5d-4e5d-831c-427d449e8fc7_isrForCrosstalkSources_996708_37.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/67d38442-8c5d-4e5d-831c-427d449e8fc7_isrForCrosstalkSources_996708_37.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8896548740f833af13704fabd59360a3 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c4b52bff-afa7-462c-9487-92f83d64912e_isrForCrosstalkSources_996714_38.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c4b52bff-afa7-462c-9487-92f83d64912e_isrForCrosstalkSources_996714_38.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4776ea43c0fc537e5aea504916325d64 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/407fd2b5-add3-476d-a012-857c6a904f51_isrForCrosstalkSources_996705_47.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/407fd2b5-add3-476d-a012-857c6a904f51_isrForCrosstalkSources_996705_47.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e3b636abd32c97dc85b2ace7aeca5d with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b76915d2-fb71-46d9-9e82-6fad76d0dbd0_isrForCrosstalkSources_996706_21.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b76915d2-fb71-46d9-9e82-6fad76d0dbd0_isrForCrosstalkSources_996706_21.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cee391da48219c41d13785f32e17ff4d with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/5bd0c617-cc93-4036-9016-c417227e7900_isrForCrosstalkSources_996712_36.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/5bd0c617-cc93-4036-9016-c417227e7900_isrForCrosstalkSources_996712_36.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0112406bbc7ab9995c70e24e4bc825d0 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/17e5649e-4d3e-4158-b492-e573e9fe37f9_isrForCrosstalkSources_996719_3.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/17e5649e-4d3e-4158-b492-e573e9fe37f9_isrForCrosstalkSources_996719_3.stderr parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f7a278314e061959bdf2e4936783b38 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/55986c27-819d-4ac7-bacc-c3b37ae4f5bc_isrForCrosstalkSources_996716_31.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/55986c27-819d-4ac7-bacc-c3b37ae4f5bc_isrForCrosstalkSources_996716_31.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd9554396687eabbe7b8d32cd1a7eadb with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/adba7068-9284-496e-ac82-124521ac607b_isrForCrosstalkSources_996716_2.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/adba7068-9284-496e-ac82-124521ac607b_isrForCrosstalkSources_996716_2.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e8c9bcfb3d2fd043b23f69933a7ccf2 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0fbe8343-0486-4cc7-a9c6-ecd486455490_isrForCrosstalkSources_996715_11.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/0fbe8343-0486-4cc7-a9c6-ecd486455490_isrForCrosstalkSources_996715_11.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20f1bb483418fc8a12aad082b7f13fe7 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ff377424-7822-45fa-8521-66f7b839b3fb_isrForCrosstalkSources_996705_46.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ff377424-7822-45fa-8521-66f7b839b3fb_isrForCrosstalkSources_996705_46.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba16e2ebe5ba8b5647b9ec705ee19804 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3237d763-9188-4fcc-86bd-aea631f76c4d_isrForCrosstalkSources_996709_42.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3237d763-9188-4fcc-86bd-aea631f76c4d_isrForCrosstalkSources_996709_42.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aef9e998bf8022fb3863b19d0151bd8 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9d086772-6474-463d-9872-f3500baca8a8_isrForCrosstalkSources_996714_17.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9d086772-6474-463d-9872-f3500baca8a8_isrForCrosstalkSources_996714_17.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2279a713484369cb8968f1c1476c2e87 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/61fded40-6164-440a-a4d8-4cdce6f6eb99_isrForCrosstalkSources_996708_16.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/61fded40-6164-440a-a4d8-4cdce6f6eb99_isrForCrosstalkSources_996708_16.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e67e440ed0284d172ca795c185b28d00 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/008649da-7d7c-4e8d-b774-00dac9ae8de9_isrForCrosstalkSources_996713_44.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/008649da-7d7c-4e8d-b774-00dac9ae8de9_isrForCrosstalkSources_996713_44.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88b7d43881968a013c2c3b097201a094 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ef8b8dbc-4b84-40f3-841f-83abf53a4a72_isrForCrosstalkSources_996713_35.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ef8b8dbc-4b84-40f3-841f-83abf53a4a72_isrForCrosstalkSources_996713_35.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45912c9c32e1cbb98e42e9dee459d15d with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8e0654c9-3466-4bc2-9ca9-415aff27d445_isrForCrosstalkSources_996717_31.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8e0654c9-3466-4bc2-9ca9-415aff27d445_isrForCrosstalkSources_996717_31.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4195efe62eb33084365cb1d961420f2 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d8d498ba-d70a-47a8-be33-5a90a024ad02_isrForCrosstalkSources_996711_55.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d8d498ba-d70a-47a8-be33-5a90a024ad02_isrForCrosstalkSources_996711_55.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba8ea5f06ce86f53b4065896cde4cffb with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/4bf875b0-9902-4571-a411-7a7975514ee1_isrForCrosstalkSources_996721_54.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/4bf875b0-9902-4571-a411-7a7975514ee1_isrForCrosstalkSources_996721_54.stderr parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 747ad59434c1d9ae45912a7fdb944457 with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/6c841711-5ac3-42c6-b643-2fbe28e92dd2_isrForCrosstalkSources_996715_2.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/6c841711-5ac3-42c6-b643-2fbe28e92dd2_isrForCrosstalkSources_996715_2.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f7f5fb76af68c6101f75a85ab3a0439 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cc3f7a54-d1ae-4819-aec8-4a8781776178_isrForCrosstalkSources_996710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cc3f7a54-d1ae-4819-aec8-4a8781776178_isrForCrosstalkSources_996710_44.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6adc84f502f4e73733e3f4afc094e828 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/bec4cedc-3ea4-48c5-a303-565f760f2448_isrForCrosstalkSources_996715_34.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/bec4cedc-3ea4-48c5-a303-565f760f2448_isrForCrosstalkSources_996715_34.stderr 2024-03-15 12:19:13 proc_lsst.multi:170 [INFO] [multi] job_id 17071325 proc_lsst.multi INFO: [multi] job_id 17071325 2024-03-15 12:19:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-15 12:19:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071325 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071325 parsl.executors.status_handling DEBUG: Launched block 137 on executor multi with job ID 17071325 parsl.executors.status_handling INFO: Allocated block ID 138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-138 2024-03-15 12:19:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 2024-03-15 12:19:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-15 12:19:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffcee6ae9d3eec433cfeb4c0afa8269b with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d9b124be-7d93-4e74-adde-1a8b5a47986d_isrForCrosstalkSources_996713_15.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/d9b124be-7d93-4e74-adde-1a8b5a47986d_isrForCrosstalkSources_996713_15.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d33f03668154fd8219d2f6b74119310c with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2340849b-eb90-445a-b3f4-7e84d73e17c7_isrForCrosstalkSources_996713_37.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2340849b-eb90-445a-b3f4-7e84d73e17c7_isrForCrosstalkSources_996713_37.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62c806e7565cf58331e0cd78a3d3203b with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/46ce4d69-3548-47d3-9259-90c064e813fa_isrForCrosstalkSources_996708_34.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/46ce4d69-3548-47d3-9259-90c064e813fa_isrForCrosstalkSources_996708_34.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44ace196f488953c45befea0fa76d0c5 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/db0b1a01-004e-43ad-9b87-c2e91c8fe4fe_isrForCrosstalkSources_996721_62.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/db0b1a01-004e-43ad-9b87-c2e91c8fe4fe_isrForCrosstalkSources_996721_62.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9607780eae031a48d846c3064098087 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/96e96f1c-cc61-4abc-a7d0-c41b092d749d_isrForCrosstalkSources_996706_37.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/96e96f1c-cc61-4abc-a7d0-c41b092d749d_isrForCrosstalkSources_996706_37.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d155ec5d8320de30a3240b346184a7e6 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/32dcc6d4-9148-4483-9021-d78604e81836_isrForCrosstalkSources_996715_40.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/32dcc6d4-9148-4483-9021-d78604e81836_isrForCrosstalkSources_996715_40.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5375e889f48b3f48d24accd27e130db9 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/97dc6c80-3715-458c-8632-bee16f4c07d5_isrForCrosstalkSources_996711_46.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/97dc6c80-3715-458c-8632-bee16f4c07d5_isrForCrosstalkSources_996711_46.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a875751753c7e94dc36e25911e21ad0 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5a30d2ac-7364-4403-bcc0-8baf75b49c24_isrForCrosstalkSources_996718_21.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5a30d2ac-7364-4403-bcc0-8baf75b49c24_isrForCrosstalkSources_996718_21.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b594c06a20b72a77238b154ad7fc5abf with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/340a6fdf-01b9-4fa0-a905-c29dd7c9df2f_isrForCrosstalkSources_996715_27.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/340a6fdf-01b9-4fa0-a905-c29dd7c9df2f_isrForCrosstalkSources_996715_27.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db2b7b7ef80b4312efe69df1d04d3b9 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2a689c76-08d3-42e5-b5bd-258c9f873d23_isrForCrosstalkSources_996718_13.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2a689c76-08d3-42e5-b5bd-258c9f873d23_isrForCrosstalkSources_996718_13.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa15d9a54cbb3274256a6e43cbbcb516 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a45e0698-b32c-4316-a48a-913b69ddc372_isrForCrosstalkSources_996706_4.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a45e0698-b32c-4316-a48a-913b69ddc372_isrForCrosstalkSources_996706_4.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c5fb232ce7def9d714d21879812385a with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/398f926b-07e0-4eb0-abfe-367df68836c8_isrForCrosstalkSources_996712_45.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/398f926b-07e0-4eb0-abfe-367df68836c8_isrForCrosstalkSources_996712_45.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59555f5113ffdf1351fffdeb9f6474e2 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/6cc05ddc-5341-4540-8d10-5c62d908721a_isrForCrosstalkSources_996716_46.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/6cc05ddc-5341-4540-8d10-5c62d908721a_isrForCrosstalkSources_996716_46.stderr 2024-03-15 12:19:14 proc_lsst.multi:170 [INFO] [multi] job_id 17071326 proc_lsst.multi INFO: [multi] job_id 17071326 2024-03-15 12:19:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-15 12:19:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071326 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071326 parsl.executors.status_handling DEBUG: Launched block 138 on executor multi with job ID 17071326 parsl.executors.status_handling INFO: Allocated block ID 139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-139 2024-03-15 12:19:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 2024-03-15 12:19:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-15 12:19:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c40df5ff509965b978ad269b9fbfcfd with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/1fef5127-1668-41b9-8c48-30772b7a5681_isrForCrosstalkSources_996706_41.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/1fef5127-1668-41b9-8c48-30772b7a5681_isrForCrosstalkSources_996706_41.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e226c50582ab7580e78cdaaae42dcf6 with result from task 550 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/b45e9b3c-c7fe-4c4f-9662-aebb1a1ec9a8_isrForCrosstalkSources_996710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/b45e9b3c-c7fe-4c4f-9662-aebb1a1ec9a8_isrForCrosstalkSources_996710_24.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e93e9e300ceea772175eda46b312c295 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/23c9b037-42e2-4a6c-bebc-7afe0c5358e1_isrForCrosstalkSources_996716_20.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/23c9b037-42e2-4a6c-bebc-7afe0c5358e1_isrForCrosstalkSources_996716_20.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea89c2345ee099d42bbb8c0413d3e700 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/954252f4-fc68-459c-9688-4d3fa375c749_isrForCrosstalkSources_996715_18.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/954252f4-fc68-459c-9688-4d3fa375c749_isrForCrosstalkSources_996715_18.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45414a8dcd9af5ca0e35daeed839c7a5 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1dfafc43-95eb-43bd-8d45-c6bd67d5ccb1_isrForCrosstalkSources_996707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1dfafc43-95eb-43bd-8d45-c6bd67d5ccb1_isrForCrosstalkSources_996707_61.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae155f61f14025524815daa079614a02 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a983dfda-47ec-4076-a970-299dc9a0c87e_isrForCrosstalkSources_996712_60.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a983dfda-47ec-4076-a970-299dc9a0c87e_isrForCrosstalkSources_996712_60.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f427f3d9dc1bff0ed1152b61bcc39f37 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f5df80d5-1aee-44d0-b5ea-cb01e47c0d9b_isrForCrosstalkSources_996711_61.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f5df80d5-1aee-44d0-b5ea-cb01e47c0d9b_isrForCrosstalkSources_996711_61.stderr 2024-03-15 12:19:15 proc_lsst.multi:170 [INFO] [multi] job_id 17071328 proc_lsst.multi INFO: [multi] job_id 17071328 2024-03-15 12:19:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-15 12:19:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071328 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071328 parsl.executors.status_handling DEBUG: Launched block 139 on executor multi with job ID 17071328 parsl.executors.status_handling INFO: Allocated block ID 140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-140 2024-03-15 12:19:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 2024-03-15 12:19:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-15 12:19:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Storing app cache entry cf711161788138f4c681b55202fa415b with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3c0c7f6b-1c24-4b9a-843a-f65e6c719f58_isrForCrosstalkSources_996705_8.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3c0c7f6b-1c24-4b9a-843a-f65e6c719f58_isrForCrosstalkSources_996705_8.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f2d54ff1309c5a4b1bdf5846b70cfc0 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e6064a2e-0ae7-44da-9bf5-045e00f5d52f_isrForCrosstalkSources_996706_5.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e6064a2e-0ae7-44da-9bf5-045e00f5d52f_isrForCrosstalkSources_996706_5.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60507e205809ccde4aea4aac84b46ce3 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4092abab-3157-4659-a6d0-2885c85d8998_isrForCrosstalkSources_996713_60.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4092abab-3157-4659-a6d0-2885c85d8998_isrForCrosstalkSources_996713_60.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 664012122c595f5420ace4d4dbbea9f0 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/0fea6995-4f61-46bb-a74f-5528445c5680_isrForCrosstalkSources_996716_35.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/0fea6995-4f61-46bb-a74f-5528445c5680_isrForCrosstalkSources_996716_35.stderr 2024-03-15 12:19:16 proc_lsst.multi:170 [INFO] [multi] job_id 17071329 proc_lsst.multi INFO: [multi] job_id 17071329 2024-03-15 12:19:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-15 12:19:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071329 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071329 parsl.executors.status_handling DEBUG: Launched block 140 on executor multi with job ID 17071329 parsl.executors.status_handling INFO: Allocated block ID 141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-141 2024-03-15 12:19:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bef3209d7cd57c4b0c5f6385e5f2e48 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c31a41fc-8cd8-4da6-8ca5-270953d54ec9_isrForCrosstalkSources_996706_50.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c31a41fc-8cd8-4da6-8ca5-270953d54ec9_isrForCrosstalkSources_996706_50.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88088904cc687fb0365325943f005402 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a62f6aee-6835-49cf-87bb-825ff2bdf5db_isrForCrosstalkSources_996712_47.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a62f6aee-6835-49cf-87bb-825ff2bdf5db_isrForCrosstalkSources_996712_47.stderr parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fcb5614335f0e71b25e11e3d2aba6c7 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/95275c3d-8ede-4e3c-a7bb-ab64545c1243_isrForCrosstalkSources_996711_53.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/95275c3d-8ede-4e3c-a7bb-ab64545c1243_isrForCrosstalkSources_996711_53.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a300d9b783d293f9f93f8f240ab70ab2 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3f220ed7-ef9d-4a99-8d2d-4481be2ab796_isrForCrosstalkSources_996707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3f220ed7-ef9d-4a99-8d2d-4481be2ab796_isrForCrosstalkSources_996707_32.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c94c546d8c6f9061dab7f618288bb405 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4adfcf7d-65f1-4aa7-984b-c1fb529386f7_isrForCrosstalkSources_996707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4adfcf7d-65f1-4aa7-984b-c1fb529386f7_isrForCrosstalkSources_996707_45.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66feeb8bda3553572878f4b2f47f7fe6 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/79ca0770-41e0-4747-a34c-523df2e1a1e9_isrForCrosstalkSources_996705_23.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/79ca0770-41e0-4747-a34c-523df2e1a1e9_isrForCrosstalkSources_996705_23.stderr 2024-03-15 12:19:16 proc_lsst.multi:170 [INFO] [multi] job_id 17071330 proc_lsst.multi INFO: [multi] job_id 17071330 2024-03-15 12:19:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-15 12:19:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071330 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071330 parsl.executors.status_handling DEBUG: Launched block 141 on executor multi with job ID 17071330 parsl.executors.status_handling INFO: Allocated block ID 142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-142 2024-03-15 12:19:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-15 12:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f156280b5700c851111239efddeb24 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e0a509c6-6e24-497d-abbb-8b7a3cfbc01f_isrForCrosstalkSources_996716_53.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e0a509c6-6e24-497d-abbb-8b7a3cfbc01f_isrForCrosstalkSources_996716_53.stderr 2024-03-15 12:19:17 proc_lsst.multi:170 [INFO] [multi] job_id 17071331 proc_lsst.multi INFO: [multi] job_id 17071331 2024-03-15 12:19:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-15 12:19:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071331 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071331 parsl.executors.status_handling DEBUG: Launched block 142 on executor multi with job ID 17071331 parsl.executors.status_handling INFO: Allocated block ID 143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-143 2024-03-15 12:19:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 2024-03-15 12:19:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-15 12:19:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 654ad555baa8a92355b41134a3ef275e with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3a2ac72d-61f8-45a7-955d-95fedc6cc6f8_isrForCrosstalkSources_996711_11.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3a2ac72d-61f8-45a7-955d-95fedc6cc6f8_isrForCrosstalkSources_996711_11.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37e09542617424651206e9c0aabf0189 with result from task 393 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4960afbb-a2ad-4f76-9c4d-d7fea7bd7f5e_isrForCrosstalkSources_996718_46.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4960afbb-a2ad-4f76-9c4d-d7fea7bd7f5e_isrForCrosstalkSources_996718_46.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22b84b1d71784a38c72b9f8f816f5612 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/ba09161d-15e5-4e1d-b8c3-4637e2a4aa00_isrForCrosstalkSources_996710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/ba09161d-15e5-4e1d-b8c3-4637e2a4aa00_isrForCrosstalkSources_996710_27.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 343bf80cc55a8873603f5da90162e605 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e777e4f2-1827-4ffc-95bc-4f6a31dedb14_isrForCrosstalkSources_996715_15.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/e777e4f2-1827-4ffc-95bc-4f6a31dedb14_isrForCrosstalkSources_996715_15.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21291f1d37a3cfb5171ca09a02e35d51 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/421523cc-fc66-4e7c-9b39-aa57a9ccbf38_isrForCrosstalkSources_996716_49.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/421523cc-fc66-4e7c-9b39-aa57a9ccbf38_isrForCrosstalkSources_996716_49.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2362a6a5227b8eb0420610f4efe0783a with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c69df5fb-d7fd-4ca7-bfc5-c1ef6f981dc2_isrForCrosstalkSources_996712_26.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c69df5fb-d7fd-4ca7-bfc5-c1ef6f981dc2_isrForCrosstalkSources_996712_26.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0f73aec8736c412ea02e896a6783ca0 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/fc0e34c2-1533-4d2e-8171-3fec59ac717a_isrForCrosstalkSources_996708_17.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/fc0e34c2-1533-4d2e-8171-3fec59ac717a_isrForCrosstalkSources_996708_17.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01318759ad168932ef44df1d8f53aa17 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7bb06742-df2d-417f-a21e-e8fb7fabd701_isrForCrosstalkSources_996721_8.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/7bb06742-df2d-417f-a21e-e8fb7fabd701_isrForCrosstalkSources_996721_8.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a139e26a54d850be1b328a7d56749b6 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5c317e2f-e97f-46b1-bbb7-6a30db1abd7d_isrForCrosstalkSources_996718_38.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/5c317e2f-e97f-46b1-bbb7-6a30db1abd7d_isrForCrosstalkSources_996718_38.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d56fc52c1f78b90687dd0e3caff6ddf with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ae7291b3-0240-49fe-a7e4-a9788e9782bb_isrForCrosstalkSources_996720_14.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ae7291b3-0240-49fe-a7e4-a9788e9782bb_isrForCrosstalkSources_996720_14.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42d5fc566b9cad9862113060b70b3ad6 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/dbac4578-beca-4f8f-86fa-7ea1f52ef638_isrForCrosstalkSources_996712_1.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/dbac4578-beca-4f8f-86fa-7ea1f52ef638_isrForCrosstalkSources_996712_1.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81b0c237e9d38c1fa9563b9870efb120 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/1301a794-3121-4097-93d5-c2850d9ef646_isrForCrosstalkSources_996714_4.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/1301a794-3121-4097-93d5-c2850d9ef646_isrForCrosstalkSources_996714_4.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c08588a6ff54fba93d5c648ef83759b4 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/4309d474-6b01-4774-b4ba-e87dca62a745_isrForCrosstalkSources_996720_19.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/4309d474-6b01-4774-b4ba-e87dca62a745_isrForCrosstalkSources_996720_19.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa9ffe8b4d0c1bf4087f08ad26a06ee with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/653edfed-c521-4b06-9bcf-99d7ca9412d2_isrForCrosstalkSources_996716_17.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/653edfed-c521-4b06-9bcf-99d7ca9412d2_isrForCrosstalkSources_996716_17.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e601e4204b80ecca975564bf6ee100a with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c7c55d1a-97d4-49d3-b97e-322860db8c16_isrForCrosstalkSources_996719_33.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c7c55d1a-97d4-49d3-b97e-322860db8c16_isrForCrosstalkSources_996719_33.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d37ffb8f67992c99a8916d411ed39023 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/038b3012-de99-402c-befb-0d132a52b982_isrForCrosstalkSources_996710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/038b3012-de99-402c-befb-0d132a52b982_isrForCrosstalkSources_996710_7.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02f7e3be0aa55f46e27b9d65432a54e4 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8e3d6682-a99b-4bc0-a595-1ed8b4a6ccbd_isrForCrosstalkSources_996720_32.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/8e3d6682-a99b-4bc0-a595-1ed8b4a6ccbd_isrForCrosstalkSources_996720_32.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90e55e3cd3af5c427c0ff3ac238b9d1c with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/63d4811d-a7ec-4615-8449-fa40d1062a3c_isrForCrosstalkSources_996710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/63d4811d-a7ec-4615-8449-fa40d1062a3c_isrForCrosstalkSources_996710_45.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9a70d3f3c2926c663c1ade52b746d5 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/2a220096-2a82-43e3-9810-a1d44b3963dc_isrForCrosstalkSources_996719_31.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/2a220096-2a82-43e3-9810-a1d44b3963dc_isrForCrosstalkSources_996719_31.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f06e0e4983f87486a4f753dfa8576465 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/bd21885c-c950-4def-b497-d81856b34b4e_isrForCrosstalkSources_996714_51.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/bd21885c-c950-4def-b497-d81856b34b4e_isrForCrosstalkSources_996714_51.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe21c4145535df686b1cecfc54425f5 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8b2c77e4-87ff-4164-9332-30ad577ee8f3_isrForCrosstalkSources_996711_59.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/8b2c77e4-87ff-4164-9332-30ad577ee8f3_isrForCrosstalkSources_996711_59.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfa1ed62d43762b41fedbf9ec677290b with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/c713270f-7045-4629-831f-a489bf2e71af_isrForCrosstalkSources_996717_16.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/c713270f-7045-4629-831f-a489bf2e71af_isrForCrosstalkSources_996717_16.stderr parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8a9575ab3c9dc86f80a6174a7fc9071 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/eb5c23ab-4076-4b1f-be39-0fe14f0a51ed_isrForCrosstalkSources_996713_5.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/eb5c23ab-4076-4b1f-be39-0fe14f0a51ed_isrForCrosstalkSources_996713_5.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf799d0cb4f0a594da0cab5c2022655f with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/5916108f-f466-4efb-b766-8f06c9148b37_isrForCrosstalkSources_996716_58.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/5916108f-f466-4efb-b766-8f06c9148b37_isrForCrosstalkSources_996716_58.stderr parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4121341fb99bcbea23bce9a859e9466 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/98cacf16-200e-47e3-b5b4-d481744c5be5_isrForCrosstalkSources_996719_26.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/98cacf16-200e-47e3-b5b4-d481744c5be5_isrForCrosstalkSources_996719_26.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09bdc55daafe2c46513e9fd23cebb065 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/71eb4b27-a458-489c-a8bd-6eeba9ca962d_isrForCrosstalkSources_996717_39.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/71eb4b27-a458-489c-a8bd-6eeba9ca962d_isrForCrosstalkSources_996717_39.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 981ae060202629c101f460065da7ca3d with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/43dad802-25f2-468b-bdb2-29a00b55c7c7_isrForCrosstalkSources_996709_11.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/43dad802-25f2-468b-bdb2-29a00b55c7c7_isrForCrosstalkSources_996709_11.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5de27ec9971fd91801c75b127cb57e99 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3ac79982-a432-4dd8-9e17-0b794e9b6457_isrForCrosstalkSources_996706_6.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3ac79982-a432-4dd8-9e17-0b794e9b6457_isrForCrosstalkSources_996706_6.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffe7f2ffe64c3a84538b3aadaf23db21 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e98e5c0a-b18f-4589-af3a-13f24a156655_isrForCrosstalkSources_996720_10.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e98e5c0a-b18f-4589-af3a-13f24a156655_isrForCrosstalkSources_996720_10.stderr parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40a4f6128378909e2c72d22f30a0ae7a with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e2d79c2d-2812-4969-89d9-7e41c6b61f9f_isrForCrosstalkSources_996706_35.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/e2d79c2d-2812-4969-89d9-7e41c6b61f9f_isrForCrosstalkSources_996706_35.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d677c15e971a9f9e7efdd39743969b with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/cf635695-b7c9-4448-b7d1-ee65b478c86f_isrForCrosstalkSources_996707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/cf635695-b7c9-4448-b7d1-ee65b478c86f_isrForCrosstalkSources_996707_44.stderr parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8b0bbe5f20fca336e6ff569d4b2afaa with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/8f87cbc8-6dfd-434a-a548-6845107a71ad_isrForCrosstalkSources_996707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/8f87cbc8-6dfd-434a-a548-6845107a71ad_isrForCrosstalkSources_996707_31.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a19bd0fc078e78e4339c6574bef44b76 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a5115635-1809-41e4-94b7-f99dad91b4eb_isrForCrosstalkSources_996712_55.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/a5115635-1809-41e4-94b7-f99dad91b4eb_isrForCrosstalkSources_996712_55.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cc4aa4c0b0a56d890787f6cab3d0a4d with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/7a91dacf-1670-45ee-8979-994ebe0d64a2_isrForCrosstalkSources_996718_48.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/7a91dacf-1670-45ee-8979-994ebe0d64a2_isrForCrosstalkSources_996718_48.stderr parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dc75a33089991b9960747fe0ca5becf with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/12d1b418-8010-475c-8c73-6498dc02514d_isrForCrosstalkSources_996719_42.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/12d1b418-8010-475c-8c73-6498dc02514d_isrForCrosstalkSources_996719_42.stderr 2024-03-15 12:19:19 proc_lsst.multi:170 [INFO] [multi] job_id 17071334 proc_lsst.multi INFO: [multi] job_id 17071334 2024-03-15 12:19:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-15 12:19:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071334 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071334 parsl.executors.status_handling DEBUG: Launched block 143 on executor multi with job ID 17071334 parsl.executors.status_handling INFO: Allocated block ID 144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-144 2024-03-15 12:19:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2d3a39bc4ab9c26efa575e281846a9a with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5992e02f-a5cb-4ea8-b1e6-3d8f29f385a7_isrForCrosstalkSources_996708_29.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5992e02f-a5cb-4ea8-b1e6-3d8f29f385a7_isrForCrosstalkSources_996708_29.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc174ef050bfc27ab3b6f47ef8317461 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8f225d1b-9659-4fda-a31f-5b62a6e43d73_isrForCrosstalkSources_996718_30.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8f225d1b-9659-4fda-a31f-5b62a6e43d73_isrForCrosstalkSources_996718_30.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53640ee03240a5a42e053f4cd974efa4 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/da740bd1-9912-410a-974e-b44bd8f3a3d5_isrForCrosstalkSources_996718_27.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/da740bd1-9912-410a-974e-b44bd8f3a3d5_isrForCrosstalkSources_996718_27.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a36d790d0eea4f777f1ff227e93fa2b with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/ca50883d-fce0-4a33-882e-cb98e65cea95_isrForCrosstalkSources_996711_60.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/ca50883d-fce0-4a33-882e-cb98e65cea95_isrForCrosstalkSources_996711_60.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 463bd84c7f4f5d9a227409f41bdac76a with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a6ebab2e-f909-4d24-bf74-869eded5319b_isrForCrosstalkSources_996719_62.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a6ebab2e-f909-4d24-bf74-869eded5319b_isrForCrosstalkSources_996719_62.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f1eacd5b132ea43900eb11d09b21687 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a375dbb7-7c92-4232-85a6-f0d5f134d143_isrForCrosstalkSources_996713_54.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a375dbb7-7c92-4232-85a6-f0d5f134d143_isrForCrosstalkSources_996713_54.stderr 2024-03-15 12:19:19 proc_lsst.multi:170 [INFO] [multi] job_id 17071335 proc_lsst.multi INFO: [multi] job_id 17071335 2024-03-15 12:19:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-15 12:19:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071335 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071335 parsl.executors.status_handling DEBUG: Launched block 144 on executor multi with job ID 17071335 parsl.executors.status_handling INFO: Allocated block ID 145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-145 2024-03-15 12:19:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-15 12:19:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e522573995363be84d29f7eaa350e2a with result from task 542 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/40e46a1b-9278-4842-86f8-46acf1832a21_isrForCrosstalkSources_996706_52.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/40e46a1b-9278-4842-86f8-46acf1832a21_isrForCrosstalkSources_996706_52.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa569bd50dac7d12b13635912ad4523a with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2f0d31fe-cb27-446a-9e5d-7ffed2892922_isrForCrosstalkSources_996718_8.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2f0d31fe-cb27-446a-9e5d-7ffed2892922_isrForCrosstalkSources_996718_8.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb723690a15e8b750b2763175d75844b with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c0060344-2870-48df-bcd4-159f42c9cbc7_isrForCrosstalkSources_996705_53.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c0060344-2870-48df-bcd4-159f42c9cbc7_isrForCrosstalkSources_996705_53.stderr parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18f79b0ba0b446d898fd12dd9040b927 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f4f5b6b0-e06c-4fe9-9d51-b98c2301fcf4_isrForCrosstalkSources_996708_33.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f4f5b6b0-e06c-4fe9-9d51-b98c2301fcf4_isrForCrosstalkSources_996708_33.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91bc4a852d118a4e2189edd14b8a3c2d with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/843fbbe4-b16a-40db-b778-499349baa574_isrForCrosstalkSources_996707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/843fbbe4-b16a-40db-b778-499349baa574_isrForCrosstalkSources_996707_21.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ffde7ac798e704d8ef62109c9d8119c with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/ec41bd70-843b-44a9-bd09-27a59a475ee2_isrForCrosstalkSources_996721_60.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/ec41bd70-843b-44a9-bd09-27a59a475ee2_isrForCrosstalkSources_996721_60.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 305e3c9bbd93fd4c2907a0bb6126c97d with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1ea9951a-a5c6-4fd1-aaa4-3543c253f5eb_isrForCrosstalkSources_996719_14.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/1ea9951a-a5c6-4fd1-aaa4-3543c253f5eb_isrForCrosstalkSources_996719_14.stderr parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b59db8b1dc32c711510163c9142093c with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/21ffcce3-00bd-4cb1-92ab-d7c13deac713_isrForCrosstalkSources_996717_23.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/21ffcce3-00bd-4cb1-92ab-d7c13deac713_isrForCrosstalkSources_996717_23.stderr parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85ffc1e22364051a59b001ba7b15ca0c with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fe7a4716-6785-485b-b1e4-f19a5a86a16b_isrForCrosstalkSources_996705_38.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/fe7a4716-6785-485b-b1e4-f19a5a86a16b_isrForCrosstalkSources_996705_38.stderr 2024-03-15 12:19:20 proc_lsst.multi:170 [INFO] [multi] job_id 17071336 proc_lsst.multi INFO: [multi] job_id 17071336 2024-03-15 12:19:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-15 12:19:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071336 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071336 parsl.executors.status_handling DEBUG: Launched block 145 on executor multi with job ID 17071336 parsl.executors.status_handling INFO: Allocated block ID 146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-146 2024-03-15 12:19:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 403540ad6df03f3770c74b206b9eb20c with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/d059ecb2-aa8b-490c-bd40-e22ca6d69c0a_isrForCrosstalkSources_996715_51.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/d059ecb2-aa8b-490c-bd40-e22ca6d69c0a_isrForCrosstalkSources_996715_51.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 638d374ef1827d61c060693568b09a42 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a99e07c4-a151-491d-985d-a542e3a4a331_isrForCrosstalkSources_996709_23.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/a99e07c4-a151-491d-985d-a542e3a4a331_isrForCrosstalkSources_996709_23.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb12e00495f5eb0106aa0ae62bea393c with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/00517c26-941a-49c6-b6fd-9cde9bdfce5e_isrForCrosstalkSources_996719_48.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/00517c26-941a-49c6-b6fd-9cde9bdfce5e_isrForCrosstalkSources_996719_48.stderr 2024-03-15 12:19:20 proc_lsst.multi:170 [INFO] [multi] job_id 17071337 proc_lsst.multi INFO: [multi] job_id 17071337 2024-03-15 12:19:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-15 12:19:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071337 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071337 parsl.executors.status_handling DEBUG: Launched block 146 on executor multi with job ID 17071337 parsl.executors.status_handling INFO: Allocated block ID 147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-147 2024-03-15 12:19:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-15 12:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 515f4b1784abcf76797eeb4c9477b2ba with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c62d1590-104a-4192-bccf-ad1532741fb9_isrForCrosstalkSources_996708_59.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/c62d1590-104a-4192-bccf-ad1532741fb9_isrForCrosstalkSources_996708_59.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c317f426bf804344447312fd8d8e4022 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/71a79d83-08a4-4c3c-a4e9-3094e68d5261_isrForCrosstalkSources_996707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/71a79d83-08a4-4c3c-a4e9-3094e68d5261_isrForCrosstalkSources_996707_39.stderr parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82fb6932534c221e4c85b51981a42e6d with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/2fc1f67d-b903-44b7-9b54-66631b30a6bd_isrForCrosstalkSources_996707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/2fc1f67d-b903-44b7-9b54-66631b30a6bd_isrForCrosstalkSources_996707_28.stderr parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e97cf3e2e4dd333a18fd7cec737d6e26 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a4a0de5d-2225-41d5-8ce1-2f3f988b959e_isrForCrosstalkSources_996720_36.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a4a0de5d-2225-41d5-8ce1-2f3f988b959e_isrForCrosstalkSources_996720_36.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b730d0e694e8ffa66f28ceb5e8c93c31 with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/db80551e-95b2-4a4c-b8e0-9bf1ad0191f1_isrForCrosstalkSources_996720_48.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/db80551e-95b2-4a4c-b8e0-9bf1ad0191f1_isrForCrosstalkSources_996720_48.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a21de2bb3c55172de90185b1ba17831 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/cf0167b8-8a61-4320-afe4-58ef44f25b6c_isrForCrosstalkSources_996706_51.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/cf0167b8-8a61-4320-afe4-58ef44f25b6c_isrForCrosstalkSources_996706_51.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8f0be95d80f65e84ff02dad994da58f with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/7da022a5-8372-4eed-8785-de5b5448f506_isrForCrosstalkSources_996719_39.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/7da022a5-8372-4eed-8785-de5b5448f506_isrForCrosstalkSources_996719_39.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f3c92ee3666f0c8a7cc74e995b5e2e6 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/a17acf34-d6ad-4660-8f26-a795d5fb09e6_isrForCrosstalkSources_996711_40.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/a17acf34-d6ad-4660-8f26-a795d5fb09e6_isrForCrosstalkSources_996711_40.stderr 2024-03-15 12:19:21 proc_lsst.multi:170 [INFO] [multi] job_id 17071338 proc_lsst.multi INFO: [multi] job_id 17071338 2024-03-15 12:19:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-15 12:19:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071338 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071338 parsl.executors.status_handling DEBUG: Launched block 147 on executor multi with job ID 17071338 parsl.executors.status_handling INFO: Allocated block ID 148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-148 2024-03-15 12:19:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4baf3a0716a6effeaa1c0584ad2de893 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d95fcbf3-a6b3-48f4-be4d-33889a047cac_isrForCrosstalkSources_996708_54.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d95fcbf3-a6b3-48f4-be4d-33889a047cac_isrForCrosstalkSources_996708_54.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10f8c52548aadf824309868fc1da51b7 with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ee95b117-fc81-4f47-b7ac-0e14431f6a2d_isrForCrosstalkSources_996716_8.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ee95b117-fc81-4f47-b7ac-0e14431f6a2d_isrForCrosstalkSources_996716_8.stderr parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b1e5586ec003602e252893018afd1d6 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/655ace0a-b5d5-4849-8774-609f96321a88_isrForCrosstalkSources_996710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/655ace0a-b5d5-4849-8774-609f96321a88_isrForCrosstalkSources_996710_13.stderr parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85615e002a94e0443c7c0d387aa06594 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/42942d77-f864-4b2d-b7ff-a5951a720380_isrForCrosstalkSources_996716_15.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/42942d77-f864-4b2d-b7ff-a5951a720380_isrForCrosstalkSources_996716_15.stderr parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 424168030e94cdc4955e5c5228aed538 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/7efa8a24-8b64-4988-a18b-c68eb101f000_isrForCrosstalkSources_996706_9.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/7efa8a24-8b64-4988-a18b-c68eb101f000_isrForCrosstalkSources_996706_9.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8223144639821c143992ba78e5dcedc with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/32040880-c4a1-4c2b-b29d-598171b662eb_isrForCrosstalkSources_996716_29.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/32040880-c4a1-4c2b-b29d-598171b662eb_isrForCrosstalkSources_996716_29.stderr 2024-03-15 12:19:21 proc_lsst.multi:170 [INFO] [multi] job_id 17071339 proc_lsst.multi INFO: [multi] job_id 17071339 2024-03-15 12:19:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-15 12:19:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071339 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071339 parsl.executors.status_handling DEBUG: Launched block 148 on executor multi with job ID 17071339 parsl.executors.status_handling INFO: Allocated block ID 149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-149 2024-03-15 12:19:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-15 12:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdac53c24fac1cfccc2d672a6f2869c1 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8d3db3b8-d11a-4a6e-a5eb-2091f2796997_isrForCrosstalkSources_996706_7.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8d3db3b8-d11a-4a6e-a5eb-2091f2796997_isrForCrosstalkSources_996706_7.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17723714af736e22e2840433cabed5d6 with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2df6d598-251a-4edf-88e2-de029052cd5c_isrForCrosstalkSources_996714_10.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2df6d598-251a-4edf-88e2-de029052cd5c_isrForCrosstalkSources_996714_10.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b37e2d02c157e1ba2bb430390cd9fa6 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/08daf021-340b-4efb-baf7-47881afa6cc4_isrForCrosstalkSources_996714_15.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/08daf021-340b-4efb-baf7-47881afa6cc4_isrForCrosstalkSources_996714_15.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89dc5617570ead19fc2c3f8e49e1746 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/0b19c764-26a7-4b70-8a99-ef0c9b93b988_isrForCrosstalkSources_996721_6.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/0b19c764-26a7-4b70-8a99-ef0c9b93b988_isrForCrosstalkSources_996721_6.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 470843161ec98bb6ed3775ac22fee734 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/e2daca34-4c88-4e50-9306-ebf9022769b0_isrForCrosstalkSources_996705_16.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/e2daca34-4c88-4e50-9306-ebf9022769b0_isrForCrosstalkSources_996705_16.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fea1721a3ae2b1d58e01c4d8abe4c6e with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ca051e81-f6e4-48cc-a187-3c7e1e9775ee_isrForCrosstalkSources_996706_47.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ca051e81-f6e4-48cc-a187-3c7e1e9775ee_isrForCrosstalkSources_996706_47.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6b7f1f7781048cb4c7b5436fa02a47d with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/a447d3fe-de3c-41e9-af10-9b6cb5439537_isrForCrosstalkSources_996721_29.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/a447d3fe-de3c-41e9-af10-9b6cb5439537_isrForCrosstalkSources_996721_29.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b058de6261279133f1c95b3b54c6b58e with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9ec5b516-e06c-45e1-a717-8046efe65545_isrForCrosstalkSources_996706_57.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9ec5b516-e06c-45e1-a717-8046efe65545_isrForCrosstalkSources_996706_57.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46e1370425878a2ef529135c19bcbecf with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ed182106-6e05-4902-ae81-45823fff1455_isrForCrosstalkSources_996705_42.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/ed182106-6e05-4902-ae81-45823fff1455_isrForCrosstalkSources_996705_42.stderr 2024-03-15 12:19:22 proc_lsst.multi:170 [INFO] [multi] job_id 17071340 proc_lsst.multi INFO: [multi] job_id 17071340 2024-03-15 12:19:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-15 12:19:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071340 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071340 parsl.executors.status_handling DEBUG: Launched block 149 on executor multi with job ID 17071340 parsl.executors.status_handling INFO: Allocated block ID 150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-150 2024-03-15 12:19:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 2024-03-15 12:19:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-15 12:19:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c512ff3f833682e3b6585151423e73d1 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/331645e6-df66-4664-8ef5-5b075ee05958_isrForCrosstalkSources_996717_62.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/331645e6-df66-4664-8ef5-5b075ee05958_isrForCrosstalkSources_996717_62.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58be5efbdd180050524ca64a5afcf538 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f780d3b1-5c25-4c95-9aec-89efcbe4bfed_isrForCrosstalkSources_996714_32.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f780d3b1-5c25-4c95-9aec-89efcbe4bfed_isrForCrosstalkSources_996714_32.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6cb279e00a0208954c2597f61083f64 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/e11f5da9-1c09-409f-9712-6fe7121cc7c0_isrForCrosstalkSources_996718_61.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/e11f5da9-1c09-409f-9712-6fe7121cc7c0_isrForCrosstalkSources_996718_61.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a18d5b987bde330dc42301d035a8f8e3 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4a904bbf-ad2a-4ff9-a25b-76e344433513_isrForCrosstalkSources_996719_25.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/4a904bbf-ad2a-4ff9-a25b-76e344433513_isrForCrosstalkSources_996719_25.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a27bc1aec8d380f14a240d1e3384607b with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/80c2b2e8-cf6a-4e30-a712-eeb37f1bb2bd_isrForCrosstalkSources_996707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/80c2b2e8-cf6a-4e30-a712-eeb37f1bb2bd_isrForCrosstalkSources_996707_43.stderr parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46ea4f78acdbaa2b18d610d91e168e5f with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/748ff9ff-33c3-47c5-a056-e42b7b9abe2b_isrForCrosstalkSources_996715_37.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/748ff9ff-33c3-47c5-a056-e42b7b9abe2b_isrForCrosstalkSources_996715_37.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9ecfe90282227c12b89bd622f8bbd5 with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5a02a25d-6231-49ee-b506-565af09c0555_isrForCrosstalkSources_996708_14.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5a02a25d-6231-49ee-b506-565af09c0555_isrForCrosstalkSources_996708_14.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e62d19041f2ab81a08a976d17330f0f3 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/0561efe3-bae1-4dd0-8c8f-56f1a590c4dc_isrForCrosstalkSources_996714_50.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/0561efe3-bae1-4dd0-8c8f-56f1a590c4dc_isrForCrosstalkSources_996714_50.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e003bc22d95033612e5a930a2f9a196 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/57498ec4-983b-4fca-8976-562268c734f2_isrForCrosstalkSources_996718_9.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/57498ec4-983b-4fca-8976-562268c734f2_isrForCrosstalkSources_996718_9.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18afeb0cb7b8193bd76be192c51bf077 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9d390578-c69e-457b-8a48-2fede5059f5b_isrForCrosstalkSources_996708_18.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9d390578-c69e-457b-8a48-2fede5059f5b_isrForCrosstalkSources_996708_18.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd8d46129de6e48243b7c4e545657e00 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e55eef4f-e75a-46f2-b1ee-ff27360b9b8a_isrForCrosstalkSources_996720_37.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e55eef4f-e75a-46f2-b1ee-ff27360b9b8a_isrForCrosstalkSources_996720_37.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcd9001da3a75e16f0fe8c3ff87e9215 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/12f204d1-e104-418e-bd9b-236667ba36ea_isrForCrosstalkSources_996719_10.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/12f204d1-e104-418e-bd9b-236667ba36ea_isrForCrosstalkSources_996719_10.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 549fb23cdcdcd48bc70f1901f2e58a5c with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8632dceb-5f03-49e2-9b34-fe7d3719f427_isrForCrosstalkSources_996721_53.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8632dceb-5f03-49e2-9b34-fe7d3719f427_isrForCrosstalkSources_996721_53.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bd9d38b9900971189842554ba6e10a5 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/aa0159ca-5e8a-4137-8c2a-38375a49097f_isrForCrosstalkSources_996705_3.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/aa0159ca-5e8a-4137-8c2a-38375a49097f_isrForCrosstalkSources_996705_3.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9683d7d00ff8821c2e3fe1e89df78fb with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/80ce1adc-f4be-47cd-b12c-47837d84ccc8_isrForCrosstalkSources_996709_49.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/80ce1adc-f4be-47cd-b12c-47837d84ccc8_isrForCrosstalkSources_996709_49.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1ca85eb7fc8cd644ae3e549815093ff with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e33e8e70-522e-4785-b6d6-5c499b99f880_isrForCrosstalkSources_996708_12.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e33e8e70-522e-4785-b6d6-5c499b99f880_isrForCrosstalkSources_996708_12.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bb1e3facd6709494ad01b08d22aa1dd with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/56d11554-844e-4c7e-8646-bf0d893058e3_isrForCrosstalkSources_996706_20.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/56d11554-844e-4c7e-8646-bf0d893058e3_isrForCrosstalkSources_996706_20.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fffe76acc63835a2d149753638f00516 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/66f0f740-891e-491f-a30c-102c35933bde_isrForCrosstalkSources_996705_39.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/66f0f740-891e-491f-a30c-102c35933bde_isrForCrosstalkSources_996705_39.stderr 2024-03-15 12:19:24 proc_lsst.multi:170 [INFO] [multi] job_id 17071341 proc_lsst.multi INFO: [multi] job_id 17071341 2024-03-15 12:19:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-15 12:19:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071341 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071341 parsl.executors.status_handling DEBUG: Launched block 150 on executor multi with job ID 17071341 parsl.executors.status_handling INFO: Allocated block ID 151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-151 2024-03-15 12:19:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 2024-03-15 12:19:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-15 12:19:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbb96d7dcdfaf742ac34ebdf293fe6c8 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/daeffda5-61e9-46a6-9e2c-9de7594a073b_isrForCrosstalkSources_996710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/daeffda5-61e9-46a6-9e2c-9de7594a073b_isrForCrosstalkSources_996710_53.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b037ed8e57a49d39f804ddc58010365a with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/101f17e7-fb96-4550-86b7-2a3b1ed78209_isrForCrosstalkSources_996720_62.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/101f17e7-fb96-4550-86b7-2a3b1ed78209_isrForCrosstalkSources_996720_62.stderr parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3b12e5851a4a22625041850b3241d7f with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7052e061-53b6-4de3-b859-f3d7fe4f36e5_isrForCrosstalkSources_996716_5.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7052e061-53b6-4de3-b859-f3d7fe4f36e5_isrForCrosstalkSources_996716_5.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7f4379be60002e8427a59acce531925 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a6067d59-0659-491a-aae6-acda34114bd7_isrForCrosstalkSources_996720_13.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a6067d59-0659-491a-aae6-acda34114bd7_isrForCrosstalkSources_996720_13.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb7a447031ea088cd157b223c5cacc95 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e5e91fc7-810c-4cab-86fc-7031bd744896_isrForCrosstalkSources_996709_56.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e5e91fc7-810c-4cab-86fc-7031bd744896_isrForCrosstalkSources_996709_56.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e15f9c61ee6cd1a5a10f1112e6c6fb3 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/993d4deb-d803-4aef-96ae-624f06d14eda_isrForCrosstalkSources_996718_40.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/993d4deb-d803-4aef-96ae-624f06d14eda_isrForCrosstalkSources_996718_40.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 173aa01fdfd73007ecca00b998a1a579 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c46ff2aa-d2aa-4d26-a932-04886f351b72_isrForCrosstalkSources_996714_22.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c46ff2aa-d2aa-4d26-a932-04886f351b72_isrForCrosstalkSources_996714_22.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e03d001b9b4d4da85faa29da2861ae with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/794a919a-d970-4f9e-8f68-1a9b36080168_isrForCrosstalkSources_996709_20.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/794a919a-d970-4f9e-8f68-1a9b36080168_isrForCrosstalkSources_996709_20.stderr parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c331dc094f06802aa46f45bd6ebb3fca with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/98c81e80-bfb6-46f6-bcd1-7691becf86a5_isrForCrosstalkSources_996719_8.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/98c81e80-bfb6-46f6-bcd1-7691becf86a5_isrForCrosstalkSources_996719_8.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 934c13ed762cd3c46c1264872950ed0e with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/b44247d4-e615-4a07-8137-e01690d5161a_isrForCrosstalkSources_996720_17.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/b44247d4-e615-4a07-8137-e01690d5161a_isrForCrosstalkSources_996720_17.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1efe0393ecb048776dabb900823408ae with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/bc1e5849-3b5e-4c27-a6df-b6828e2be4b1_isrForCrosstalkSources_996720_25.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/bc1e5849-3b5e-4c27-a6df-b6828e2be4b1_isrForCrosstalkSources_996720_25.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17e41db94c7bf6e710d7aec56e8c57bd with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/553bc1e9-a317-4486-9530-cd4167cd0a2c_isrForCrosstalkSources_996718_12.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/553bc1e9-a317-4486-9530-cd4167cd0a2c_isrForCrosstalkSources_996718_12.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c77e61ba22bbe364278efe10e64f8514 with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/d0f7d54c-819b-4a68-b01e-64e9413fc175_isrForCrosstalkSources_996720_5.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/d0f7d54c-819b-4a68-b01e-64e9413fc175_isrForCrosstalkSources_996720_5.stderr parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b96bc0d8485d50368c388af90d359e39 with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/32ec9c26-783b-42bc-8465-112d0c915838_isrForCrosstalkSources_996709_1.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/32ec9c26-783b-42bc-8465-112d0c915838_isrForCrosstalkSources_996709_1.stderr 2024-03-15 12:19:25 proc_lsst.multi:170 [INFO] [multi] job_id 17071342 proc_lsst.multi INFO: [multi] job_id 17071342 2024-03-15 12:19:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-15 12:19:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071342 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071342 parsl.executors.status_handling DEBUG: Launched block 151 on executor multi with job ID 17071342 parsl.executors.status_handling INFO: Allocated block ID 152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-152 2024-03-15 12:19:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 2024-03-15 12:19:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-15 12:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e92659536f3047daac88bfbff6f137ca with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/8abf1a8b-97f4-48ab-85ee-e570c9e220d6_isrForCrosstalkSources_996719_7.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/8abf1a8b-97f4-48ab-85ee-e570c9e220d6_isrForCrosstalkSources_996719_7.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6e975f5ad9251ea5861c37ab4f0570f with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/299e6579-ee3e-42b6-bc24-113e7ad490bd_isrForCrosstalkSources_996707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/299e6579-ee3e-42b6-bc24-113e7ad490bd_isrForCrosstalkSources_996707_7.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3da1d0efec3785e0b9e2dfcbd2c86e4d with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/2b74e048-3337-4f5a-8fbe-bab6d1073bc2_isrForCrosstalkSources_996705_22.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/2b74e048-3337-4f5a-8fbe-bab6d1073bc2_isrForCrosstalkSources_996705_22.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f61b46954978cabf61876b1ad8c3ea8 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/a71f163d-cadf-4113-acac-b6d9e10e4c0d_isrForCrosstalkSources_996716_37.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/a71f163d-cadf-4113-acac-b6d9e10e4c0d_isrForCrosstalkSources_996716_37.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff973a09c6ee3f94e1441f3f65f0adc2 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7dbb9aed-fd83-4943-a451-08ffac6aaab2_isrForCrosstalkSources_996716_59.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/7dbb9aed-fd83-4943-a451-08ffac6aaab2_isrForCrosstalkSources_996716_59.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cc0038be8bc2ccb729f623d0adf05b0 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/391a39d8-4e27-4d6f-9bab-a4a60228d164_isrForCrosstalkSources_996716_60.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/391a39d8-4e27-4d6f-9bab-a4a60228d164_isrForCrosstalkSources_996716_60.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f361920006c4c887bdd00b258e9a4f0 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/af10fd39-c53a-4ac7-8174-9cc939d89ffe_isrForCrosstalkSources_996709_59.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/af10fd39-c53a-4ac7-8174-9cc939d89ffe_isrForCrosstalkSources_996709_59.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fec45fc45a387cc3bc4b7f343c61c67 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fe8851e0-0df0-4f10-8ce1-b37f35602c18_isrForCrosstalkSources_996721_50.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/fe8851e0-0df0-4f10-8ce1-b37f35602c18_isrForCrosstalkSources_996721_50.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1a512aee1ed3b62d1df7aeb588b81c6 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a5180114-a82d-4da4-a52a-4db6e6a28084_isrForCrosstalkSources_996720_34.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a5180114-a82d-4da4-a52a-4db6e6a28084_isrForCrosstalkSources_996720_34.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff113a0bb7f5a64590aa2f8710d3cfd2 with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7539b546-5023-4df1-a385-21b303350e96_isrForCrosstalkSources_996715_32.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7539b546-5023-4df1-a385-21b303350e96_isrForCrosstalkSources_996715_32.stderr parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d1e68cbed8b98d4f07be2b211b761be with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7ab68d7a-58d6-43a7-b08f-6aa9df1b28e6_isrForCrosstalkSources_996715_10.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7ab68d7a-58d6-43a7-b08f-6aa9df1b28e6_isrForCrosstalkSources_996715_10.stderr parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28a50661907adc4c8f65b709fd474905 with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/9d776dc8-a68a-4064-ae7f-ee253de30b22_isrForCrosstalkSources_996705_25.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/9d776dc8-a68a-4064-ae7f-ee253de30b22_isrForCrosstalkSources_996705_25.stderr parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9a27a021d1bab4014aaf98a605f6d45 with result from task 836 2024-03-15 12:19:26 proc_lsst.multi:170 [INFO] [multi] job_id 17071343 proc_lsst.multi INFO: [multi] job_id 17071343 2024-03-15 12:19:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-15 12:19:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071343 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071343 parsl.executors.status_handling DEBUG: Launched block 152 on executor multi with job ID 17071343 parsl.executors.status_handling INFO: Allocated block ID 153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-153 2024-03-15 12:19:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 2024-03-15 12:19:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-15 12:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ddccbf67-7269-43a3-8850-4c1ae0892107_isrForCrosstalkSources_996713_2.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ddccbf67-7269-43a3-8850-4c1ae0892107_isrForCrosstalkSources_996713_2.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2719054b9ccb5f21db6204fb1ad2a1c with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8d551041-da58-40e8-a455-6b2429ccfd8d_isrForCrosstalkSources_996715_59.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8d551041-da58-40e8-a455-6b2429ccfd8d_isrForCrosstalkSources_996715_59.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df74f22257bd7b97d0325e7deabd656d with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/fa9fad0f-ea6d-4faa-b3b1-e642bd687f52_isrForCrosstalkSources_996711_31.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/fa9fad0f-ea6d-4faa-b3b1-e642bd687f52_isrForCrosstalkSources_996711_31.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef6a215a3bd85774ede3d393b9b2f014 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/9c670123-9eee-465c-9ee4-74f61f033383_isrForCrosstalkSources_996715_45.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/9c670123-9eee-465c-9ee4-74f61f033383_isrForCrosstalkSources_996715_45.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd9ae7737396e5c33ed6ddd627bacee with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/50d6fe96-5f5c-4855-aa4f-74daf7a4abae_isrForCrosstalkSources_996715_21.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/50d6fe96-5f5c-4855-aa4f-74daf7a4abae_isrForCrosstalkSources_996715_21.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f44753281c73dfd94b3453371c74b751 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/49e8aee2-d99d-4237-9dfa-f73749a40324_isrForCrosstalkSources_996710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/49e8aee2-d99d-4237-9dfa-f73749a40324_isrForCrosstalkSources_996710_59.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 803bfed5a57ad6d2473ea34aabb73557 with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/078459c7-6600-479e-ad0b-1dc02bbb11aa_isrForCrosstalkSources_996721_12.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/078459c7-6600-479e-ad0b-1dc02bbb11aa_isrForCrosstalkSources_996721_12.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry debd352a6f7f31720b0e02486b22da1c with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/773f59b6-2997-4f7f-a8e4-6f960a92325a_isrForCrosstalkSources_996721_4.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/773f59b6-2997-4f7f-a8e4-6f960a92325a_isrForCrosstalkSources_996721_4.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b140101337b7607526b785de60cd8c5 with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/aad43998-9116-4a61-8460-c699431bcb57_isrForCrosstalkSources_996714_52.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/aad43998-9116-4a61-8460-c699431bcb57_isrForCrosstalkSources_996714_52.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa648e53c54f96f04e2c8b0e29a18f27 with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c51a8264-5506-41a8-9494-598e896160a6_isrForCrosstalkSources_996709_5.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c51a8264-5506-41a8-9494-598e896160a6_isrForCrosstalkSources_996709_5.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 855ebaf974744ccb0933dfcfcb62e757 with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1b8f528f-206f-4d0a-b4d1-ec297a1075b3_isrForCrosstalkSources_996705_10.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1b8f528f-206f-4d0a-b4d1-ec297a1075b3_isrForCrosstalkSources_996705_10.stderr parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 393769e925eb1f831d3ddc6c5464f5dc with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5cc9c406-ff35-42a6-9d2e-658d6fc0c8d2_isrForCrosstalkSources_996710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5cc9c406-ff35-42a6-9d2e-658d6fc0c8d2_isrForCrosstalkSources_996710_35.stderr parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 157f5f2b170cec22372a18f1ef9baa68 with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/87e67bc2-8a0d-43eb-b067-7a62180ede8f_isrForCrosstalkSources_996713_1.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/87e67bc2-8a0d-43eb-b067-7a62180ede8f_isrForCrosstalkSources_996713_1.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 386fdc7ccec917f1fac0539b8a932cd4 with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3bffb26e-395c-4c9c-b1bc-dad3dd7b6985_isrForCrosstalkSources_996706_40.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3bffb26e-395c-4c9c-b1bc-dad3dd7b6985_isrForCrosstalkSources_996706_40.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4eb2ea683b2c856fbcc78a75d9ba63c with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/97bec4ae-97e9-40bf-95f5-1bf40ac30025_isrForCrosstalkSources_996712_24.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/97bec4ae-97e9-40bf-95f5-1bf40ac30025_isrForCrosstalkSources_996712_24.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19390cf5e929156eb228d35b9614d8bf with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ff3bc849-2ccb-435b-8330-880e05123253_isrForCrosstalkSources_996720_60.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/ff3bc849-2ccb-435b-8330-880e05123253_isrForCrosstalkSources_996720_60.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a6513ed573db7fea9d66b156d8ccaab with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/976aa0e6-1384-49b4-b5f0-2730e333a7a4_isrForCrosstalkSources_996719_38.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/976aa0e6-1384-49b4-b5f0-2730e333a7a4_isrForCrosstalkSources_996719_38.stderr parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 238c44bafa16cc460953b18e04f39cd8 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/42027b16-8f9b-44dc-881e-40194b05a6cc_isrForCrosstalkSources_996712_32.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/42027b16-8f9b-44dc-881e-40194b05a6cc_isrForCrosstalkSources_996712_32.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22ac8eb976c38c04cd4ef372bfd9f45f with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b1753b7c-8bf9-44cd-9599-e249a37b0fe2_isrForCrosstalkSources_996713_14.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b1753b7c-8bf9-44cd-9599-e249a37b0fe2_isrForCrosstalkSources_996713_14.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3d0865f84744c33ec663ce9d059dff0 with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9ffccccf-c379-45b6-b50f-46ff71b9d0cb_isrForCrosstalkSources_996714_42.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9ffccccf-c379-45b6-b50f-46ff71b9d0cb_isrForCrosstalkSources_996714_42.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8639789ff64d7c8ae2eeec904f437dd1 with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/39a495ca-a749-45fa-aec7-e6795f463e60_isrForCrosstalkSources_996705_60.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/39a495ca-a749-45fa-aec7-e6795f463e60_isrForCrosstalkSources_996705_60.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47579f8500c3316e2cc8851a1fc91d9c with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/001bee0a-a7b3-48d9-85c2-e764a3ba5fad_isrForCrosstalkSources_996715_56.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/001bee0a-a7b3-48d9-85c2-e764a3ba5fad_isrForCrosstalkSources_996715_56.stderr parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 552f425a36c2182aa88731dc76c04683 with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/a5cb819c-6378-46cc-86ac-b9e13cf3d788_isrForCrosstalkSources_996721_52.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/a5cb819c-6378-46cc-86ac-b9e13cf3d788_isrForCrosstalkSources_996721_52.stderr parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f833e027aa3d0f7752eff519637ba397 with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a28c8baa-4e10-4393-a8be-0b00cc6be964_isrForCrosstalkSources_996705_33.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a28c8baa-4e10-4393-a8be-0b00cc6be964_isrForCrosstalkSources_996705_33.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06ecfac5c7d577b3d8f92c250d2d232b with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/1dc29edd-0c13-436e-b9f4-5fb6cdab8290_isrForCrosstalkSources_996720_43.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/1dc29edd-0c13-436e-b9f4-5fb6cdab8290_isrForCrosstalkSources_996720_43.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebd1c6f02cda08ee8ad1761e51bbad3d with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/b2925e7e-8aad-419f-acc2-d87d360dc15c_isrForCrosstalkSources_996714_58.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/b2925e7e-8aad-419f-acc2-d87d360dc15c_isrForCrosstalkSources_996714_58.stderr parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8493a9be69e0da6f31f9fb44c3ecb35e with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/74b81d51-62cb-45e5-9b9d-fe32eec62686_isrForCrosstalkSources_996715_28.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/74b81d51-62cb-45e5-9b9d-fe32eec62686_isrForCrosstalkSources_996715_28.stderr parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22db113c9766b513d7de7d5fd5852623 with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6e702b3a-5717-40bb-94f7-bb8034129010_isrForCrosstalkSources_996710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6e702b3a-5717-40bb-94f7-bb8034129010_isrForCrosstalkSources_996710_20.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cd6e6bf3a3e489b1723e792363a2f21 with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/476ce46b-2baa-4a9c-8a6f-4b1c7605398f_isrForCrosstalkSources_996710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/476ce46b-2baa-4a9c-8a6f-4b1c7605398f_isrForCrosstalkSources_996710_15.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f579cbb9e58a3a77f320ed41b83af34b with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4e9d5ad9-5508-45f2-9446-e52c7a48f685_isrForCrosstalkSources_996707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4e9d5ad9-5508-45f2-9446-e52c7a48f685_isrForCrosstalkSources_996707_27.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3143d30f6ad5c81432058af4ff5d4a8e with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/77c3154f-5c06-453c-b751-359ad2dc2434_isrForCrosstalkSources_996714_46.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/77c3154f-5c06-453c-b751-359ad2dc2434_isrForCrosstalkSources_996714_46.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfd069be8041ee1e22ba7ee0be38224d with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b02c220f-f7d5-4f5d-9aac-7bc075c6b319_isrForCrosstalkSources_996713_8.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b02c220f-f7d5-4f5d-9aac-7bc075c6b319_isrForCrosstalkSources_996713_8.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3464b36e615176125fc3301e67d5cbea with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/3d0d4505-e752-4519-80d8-45b1999dfd60_isrForCrosstalkSources_996714_61.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/3d0d4505-e752-4519-80d8-45b1999dfd60_isrForCrosstalkSources_996714_61.stderr parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ae9b66c3f8a146e9025614d1e7455d3 with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2faff2e1-06c3-41e0-bcdc-0b9d4fc7872c_isrForCrosstalkSources_996718_14.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/2faff2e1-06c3-41e0-bcdc-0b9d4fc7872c_isrForCrosstalkSources_996718_14.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b14d8d0ef16c467ecf8d2abba4fba2c8 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c29c46e6-e215-4bdd-b857-9b94dfe3e725_isrForCrosstalkSources_996714_56.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c29c46e6-e215-4bdd-b857-9b94dfe3e725_isrForCrosstalkSources_996714_56.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 989101baf2707aacf3b80eee10937e26 with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/eb722317-9f88-471d-89ef-5aeb79b7c506_isrForCrosstalkSources_996708_39.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/eb722317-9f88-471d-89ef-5aeb79b7c506_isrForCrosstalkSources_996708_39.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f855249953ebe076aa6a222ad5f8265 with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/18bd4882-35b0-4c85-a73a-87bc61f021f3_isrForCrosstalkSources_996713_21.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/18bd4882-35b0-4c85-a73a-87bc61f021f3_isrForCrosstalkSources_996713_21.stderr 2024-03-15 12:19:28 proc_lsst.multi:170 [INFO] [multi] job_id 17071344 proc_lsst.multi INFO: [multi] job_id 17071344 2024-03-15 12:19:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-15 12:19:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071344 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071344 parsl.executors.status_handling DEBUG: Launched block 153 on executor multi with job ID 17071344 parsl.executors.status_handling INFO: Allocated block ID 154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-154 2024-03-15 12:19:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 2024-03-15 12:19:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-15 12:19:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ce699a57bdf63792a73c000f6a7c5b7 with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d404d5d3-6215-450f-892c-3ed3e0d98e24_isrForCrosstalkSources_996708_19.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d404d5d3-6215-450f-892c-3ed3e0d98e24_isrForCrosstalkSources_996708_19.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83ca41af3b693e7053021f87391cee69 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/1194829d-3506-4d55-98e7-70d6fd31da97_isrForCrosstalkSources_996716_43.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/1194829d-3506-4d55-98e7-70d6fd31da97_isrForCrosstalkSources_996716_43.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aebbabc6289a9ae47c32c2016a17b9b with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d59f39e1-c362-40ae-bc57-f3a030cde833_isrForCrosstalkSources_996711_37.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/d59f39e1-c362-40ae-bc57-f3a030cde833_isrForCrosstalkSources_996711_37.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e9889b16fecd52577488c4e1341f08 with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/834cdccf-6c93-4dc7-a9b8-92e4c6e5afb1_isrForCrosstalkSources_996706_32.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/834cdccf-6c93-4dc7-a9b8-92e4c6e5afb1_isrForCrosstalkSources_996706_32.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63fb649516fb02d252f3735f9bbbe78f with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5b1a94a9-f7d1-4149-bee9-e7bf6f1dd127_isrForCrosstalkSources_996721_36.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5b1a94a9-f7d1-4149-bee9-e7bf6f1dd127_isrForCrosstalkSources_996721_36.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6db452b9478425cb5ddd382fbcc0088c with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/25d9eabc-da48-486d-b0ca-0b436ac51816_isrForCrosstalkSources_996707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/25d9eabc-da48-486d-b0ca-0b436ac51816_isrForCrosstalkSources_996707_60.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31b747048f03a93f3458c395d5987608 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e43ef72f-ffbb-47c8-93d9-c0268739c512_isrForCrosstalkSources_996708_31.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/e43ef72f-ffbb-47c8-93d9-c0268739c512_isrForCrosstalkSources_996708_31.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3f3b6d25a0df6806926c1c7dc7af824 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/80c66cbd-47df-4cd9-9b62-ac5d88295495_isrForCrosstalkSources_996713_32.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/80c66cbd-47df-4cd9-9b62-ac5d88295495_isrForCrosstalkSources_996713_32.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09ad31e1b5031cf02a159d68484c57f4 with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/0c203e92-afc3-4f3a-be98-eee6f7cc19bc_isrForCrosstalkSources_996721_45.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/0c203e92-afc3-4f3a-be98-eee6f7cc19bc_isrForCrosstalkSources_996721_45.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e86d808f9aaf55eacfb5a82513ba24 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/a084ff6f-22fc-44cc-9654-712e15877cfc_isrForCrosstalkSources_996715_43.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/a084ff6f-22fc-44cc-9654-712e15877cfc_isrForCrosstalkSources_996715_43.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dff1af7f5b3047e67252d52fe9562ccf with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/51099594-147d-4d80-918a-4551afb4e76b_isrForCrosstalkSources_996707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/51099594-147d-4d80-918a-4551afb4e76b_isrForCrosstalkSources_996707_8.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5eba88e060f6f2983ccb39ae7b02f602 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/60529b3c-29c3-4a0b-80d0-883fa78c7b1b_isrForCrosstalkSources_996717_57.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/60529b3c-29c3-4a0b-80d0-883fa78c7b1b_isrForCrosstalkSources_996717_57.stderr parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c96b2ab9c78464e407a95276e1567e13 with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/939ba26e-e659-4b2b-8406-543e3596047f_isrForCrosstalkSources_996714_11.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/939ba26e-e659-4b2b-8406-543e3596047f_isrForCrosstalkSources_996714_11.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb3e8b7851f9b6b6eb0ab50f9202953d with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/272f7c1a-3230-4786-9811-8ff52175b9e1_isrForCrosstalkSources_996709_55.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/272f7c1a-3230-4786-9811-8ff52175b9e1_isrForCrosstalkSources_996709_55.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adbe2319648468222925d7a09516f1bd with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/0aa61b9b-f766-4830-bff7-1bc4ad368d35_isrForCrosstalkSources_996709_25.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/0aa61b9b-f766-4830-bff7-1bc4ad368d35_isrForCrosstalkSources_996709_25.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 089e9b3488d72db3692bf065e447f9e3 with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/1628241e-5feb-454b-b12f-8abe1dac1bce_isrForCrosstalkSources_996715_62.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/1628241e-5feb-454b-b12f-8abe1dac1bce_isrForCrosstalkSources_996715_62.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e031032bb955d078994df683274f1856 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a691ce94-e350-4f9a-a73b-3067a5dd590b_isrForCrosstalkSources_996705_26.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/a691ce94-e350-4f9a-a73b-3067a5dd590b_isrForCrosstalkSources_996705_26.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f30a2ed2c9cad45e448ff11d6fd5c10 with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/aa3d520a-f3eb-4ea5-8e5a-5f207f97e0ee_isrForCrosstalkSources_996708_28.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/aa3d520a-f3eb-4ea5-8e5a-5f207f97e0ee_isrForCrosstalkSources_996708_28.stderr parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7046891d2332e7ac1a65efa692af023a with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c93b7a70-bb95-4139-ba0b-bae8fa39544c_isrForCrosstalkSources_996712_10.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c93b7a70-bb95-4139-ba0b-bae8fa39544c_isrForCrosstalkSources_996712_10.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fdb663313f34b79854802bbd6e45260 with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/7e93dbda-5db7-4da1-8c9b-011d533bc97e_isrForCrosstalkSources_996720_24.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/7e93dbda-5db7-4da1-8c9b-011d533bc97e_isrForCrosstalkSources_996720_24.stderr parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d71e69dbd21b8656d6b3e80638aeceb8 with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3faaa108-90a5-47fb-9f2a-ea6a8e35eebc_isrForCrosstalkSources_996720_46.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3faaa108-90a5-47fb-9f2a-ea6a8e35eebc_isrForCrosstalkSources_996720_46.stderr parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abea6aa712196f43f86bf66095257eac with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/7a4176c3-c63d-4b1e-a085-fc92378db4f2_isrForCrosstalkSources_996718_18.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/7a4176c3-c63d-4b1e-a085-fc92378db4f2_isrForCrosstalkSources_996718_18.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed8fc6c7491ec491d6c005039ee7e8d3 with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cc20d248-faca-40ca-ac5a-505c26293035_isrForCrosstalkSources_996710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cc20d248-faca-40ca-ac5a-505c26293035_isrForCrosstalkSources_996710_40.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34edc04bcafdc9249c28d96911d68db4 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4b2317fa-9bd5-43c5-a22d-e8458af40867_isrForCrosstalkSources_996713_56.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4b2317fa-9bd5-43c5-a22d-e8458af40867_isrForCrosstalkSources_996713_56.stderr parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9376947155bcb865950b7855ac700f7e with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/8c56dd30-9eb1-40c5-81ab-9a2d293a97f9_isrForCrosstalkSources_996709_14.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/8c56dd30-9eb1-40c5-81ab-9a2d293a97f9_isrForCrosstalkSources_996709_14.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1b1d427354e58e026ef6b59710fa5b3 with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/5c9daf33-d02b-49be-970b-7972268347f9_isrForCrosstalkSources_996705_29.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/5c9daf33-d02b-49be-970b-7972268347f9_isrForCrosstalkSources_996705_29.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 743cff5eadabc3f0457664b1af901979 with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/3276a900-49a2-495c-bdaf-dd72d1a263f1_isrForCrosstalkSources_996710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/3276a900-49a2-495c-bdaf-dd72d1a263f1_isrForCrosstalkSources_996710_16.stderr parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 339969e1afaf70f675cf17770ff696c1 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c61e760e-3ac6-493b-84ca-8e0b871a5cae_isrForCrosstalkSources_996713_20.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c61e760e-3ac6-493b-84ca-8e0b871a5cae_isrForCrosstalkSources_996713_20.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 750b08a214cc4ed341724c557e66b986 with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e58bfe34-4749-46df-b0c3-79852b1126ef_isrForCrosstalkSources_996717_24.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e58bfe34-4749-46df-b0c3-79852b1126ef_isrForCrosstalkSources_996717_24.stderr parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f263dfbe82ebd6611908a873d9f8933e with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4df4f8c8-df2f-46ce-aebf-5e9a56ee985d_isrForCrosstalkSources_996714_1.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4df4f8c8-df2f-46ce-aebf-5e9a56ee985d_isrForCrosstalkSources_996714_1.stderr parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84ee4740345771a598dd1d817d242146 with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2066d0fc-d081-423e-8503-325ca900a448_isrForCrosstalkSources_996712_48.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/2066d0fc-d081-423e-8503-325ca900a448_isrForCrosstalkSources_996712_48.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 625abb7716f0cebc292ddc59c0819f21 with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/3d87c29e-5cf8-4f89-85a6-27c0127ab9a7_isrForCrosstalkSources_996721_57.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/3d87c29e-5cf8-4f89-85a6-27c0127ab9a7_isrForCrosstalkSources_996721_57.stderr parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc03bfc2283be649aca5a40f5b125c11 with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/9b2cbc86-b058-4bc0-904c-0001327b9fdc_isrForCrosstalkSources_996720_38.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/9b2cbc86-b058-4bc0-904c-0001327b9fdc_isrForCrosstalkSources_996720_38.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 913567e98786fd1b0d27702304ba7305 with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2ceac0a4-93d6-4791-8cdd-b36779bd7996_isrForCrosstalkSources_996710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/2ceac0a4-93d6-4791-8cdd-b36779bd7996_isrForCrosstalkSources_996710_11.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3efcc460f831da09ff33a02317341941 with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/e0c95001-7db4-49b2-a616-f4e6c362b039_isrForCrosstalkSources_996719_44.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/e0c95001-7db4-49b2-a616-f4e6c362b039_isrForCrosstalkSources_996719_44.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a83ddfc2f2dffd43a20e29a0d97c7cac with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f46c1d50-a0fd-4601-ba56-2117db0086b3_isrForCrosstalkSources_996714_28.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/f46c1d50-a0fd-4601-ba56-2117db0086b3_isrForCrosstalkSources_996714_28.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c0a4bb74cdd4f75a0e1718d7b434987 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/68727362-09b9-4f1a-b09a-15110e22edd3_isrForCrosstalkSources_996717_19.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/68727362-09b9-4f1a-b09a-15110e22edd3_isrForCrosstalkSources_996717_19.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4330b750752ace925aca9640efe49598 with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c60b6fc5-1833-4522-ace4-67de7a4ba4c8_isrForCrosstalkSources_996712_30.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c60b6fc5-1833-4522-ace4-67de7a4ba4c8_isrForCrosstalkSources_996712_30.stderr 2024-03-15 12:19:30 proc_lsst.multi:170 [INFO] [multi] job_id 17071345 proc_lsst.multi INFO: [multi] job_id 17071345 2024-03-15 12:19:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-15 12:19:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071345 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071345 parsl.executors.status_handling DEBUG: Launched block 154 on executor multi with job ID 17071345 parsl.executors.status_handling INFO: Allocated block ID 155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-155 2024-03-15 12:19:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2c6d563422b378213a4e3bc46b36f9d with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/cf4221ac-ecb0-41f4-993b-71a5f384fe7a_isrForCrosstalkSources_996712_46.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/cf4221ac-ecb0-41f4-993b-71a5f384fe7a_isrForCrosstalkSources_996712_46.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bf657e2d8426e9c434487668a85180e with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/11cfe9d1-02e4-4afc-8b51-44bdd720ce7d_isrForCrosstalkSources_996705_13.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/11cfe9d1-02e4-4afc-8b51-44bdd720ce7d_isrForCrosstalkSources_996705_13.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ee765c1d93b7712bd7c166156f65fbd with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/3cbf7ff4-ccc0-4b07-8422-b3dd7727de21_isrForCrosstalkSources_996716_33.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/3cbf7ff4-ccc0-4b07-8422-b3dd7727de21_isrForCrosstalkSources_996716_33.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0820280fbfcb9c8e9e8ab092235091ed with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5fdd79b8-aec1-4eb8-8b01-a87aa55a18c8_isrForCrosstalkSources_996720_55.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/5fdd79b8-aec1-4eb8-8b01-a87aa55a18c8_isrForCrosstalkSources_996720_55.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f755c6caca869595d21094d7ee8a461 with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/025a36e4-faac-4c7c-9210-b8bd1dbfaac9_isrForCrosstalkSources_996716_4.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/025a36e4-faac-4c7c-9210-b8bd1dbfaac9_isrForCrosstalkSources_996716_4.stderr parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9ae595dc0fecdc4f37c635f3a93c770 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/2a9fa2e7-f555-4ad0-af09-583ed06a3fc4_isrForCrosstalkSources_996719_40.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/2a9fa2e7-f555-4ad0-af09-583ed06a3fc4_isrForCrosstalkSources_996719_40.stderr parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b76e94e8d95126bc61694d5b3cfe5ed with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/52b8a47a-7182-4269-b49d-8245a056738b_isrForCrosstalkSources_996719_5.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/52b8a47a-7182-4269-b49d-8245a056738b_isrForCrosstalkSources_996719_5.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32adc84847064dc0c9e1899e6b28d6e4 with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/c443aadc-9293-4138-8c0e-1b86c8977c70_isrForCrosstalkSources_996716_50.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/c443aadc-9293-4138-8c0e-1b86c8977c70_isrForCrosstalkSources_996716_50.stderr 2024-03-15 12:19:30 proc_lsst.multi:170 [INFO] [multi] job_id 17071346 proc_lsst.multi INFO: [multi] job_id 17071346 2024-03-15 12:19:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-15 12:19:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071346 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071346 parsl.executors.status_handling DEBUG: Launched block 155 on executor multi with job ID 17071346 parsl.executors.status_handling INFO: Allocated block ID 156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-156 2024-03-15 12:19:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-15 12:19:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e652b3171cd920c16bbada53aa746e17 with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/0f233dea-4342-421d-985f-dd0d539e55c6_isrForCrosstalkSources_996708_10.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/0f233dea-4342-421d-985f-dd0d539e55c6_isrForCrosstalkSources_996708_10.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e54661ebaac4129d6e8f6ce3c95007b6 with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/0ebe804f-b927-4d8d-8629-651a4fd1a256_isrForCrosstalkSources_996711_20.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/0ebe804f-b927-4d8d-8629-651a4fd1a256_isrForCrosstalkSources_996711_20.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 790d23b0a6d04883521f1f549bf36fc4 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/584a0d2a-b578-41e9-9451-b3ce15e666aa_isrForCrosstalkSources_996707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/584a0d2a-b578-41e9-9451-b3ce15e666aa_isrForCrosstalkSources_996707_16.stderr parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75e06b2154893e5a66552b8a23b3d64e with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/addf3674-b1f6-46ce-adf9-0acb9570081f_isrForCrosstalkSources_996708_3.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/addf3674-b1f6-46ce-adf9-0acb9570081f_isrForCrosstalkSources_996708_3.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 197d041cf3929b4b043679b56d4547f4 with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d0c8cca8-7483-48f9-9d33-b26a7414f6d5_isrForCrosstalkSources_996708_2.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/d0c8cca8-7483-48f9-9d33-b26a7414f6d5_isrForCrosstalkSources_996708_2.stderr parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 950daddc7929bd81e7c9a31377345a84 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/818eb3c1-546b-49e5-bf0b-d545ac1dc791_isrForCrosstalkSources_996707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/818eb3c1-546b-49e5-bf0b-d545ac1dc791_isrForCrosstalkSources_996707_38.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 756d151fd046d14003cdaa0cd5ed4c66 with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/48ca20fa-dc5e-4fa1-abee-357de42eb19e_isrForCrosstalkSources_996708_49.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/48ca20fa-dc5e-4fa1-abee-357de42eb19e_isrForCrosstalkSources_996708_49.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f49c3e51ea0603c3fdfd4cdbc7f54e2 with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/610ef7c2-26c6-432c-8c4c-33a16c3d04db_isrForCrosstalkSources_996705_58.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/610ef7c2-26c6-432c-8c4c-33a16c3d04db_isrForCrosstalkSources_996705_58.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cee660d14418cb438a0347f78a4f5e9f with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/d64a4867-eaf4-4807-8895-2ad7f6b55d95_isrForCrosstalkSources_996716_57.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/d64a4867-eaf4-4807-8895-2ad7f6b55d95_isrForCrosstalkSources_996716_57.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69837b045f989670093c7af2a31c922a with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/6a865d33-4cb0-4725-bc8d-dda0bcc02d71_isrForCrosstalkSources_996705_7.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/6a865d33-4cb0-4725-bc8d-dda0bcc02d71_isrForCrosstalkSources_996705_7.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbba4cf637d6e91025f07ccba3422636 with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/511eae03-067a-40bd-8677-da3f8db8504f_isrForCrosstalkSources_996710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/511eae03-067a-40bd-8677-da3f8db8504f_isrForCrosstalkSources_996710_49.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 648b7f2e4410e5f7d664222dd9ea12fc with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/aa56a0bb-2c11-4d84-960e-12fdd5618a83_isrForCrosstalkSources_996706_23.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/aa56a0bb-2c11-4d84-960e-12fdd5618a83_isrForCrosstalkSources_996706_23.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6a1b87e3be38763523cb423a6f0ed33 with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks 2024-03-15 12:19:31 proc_lsst.multi:170 [INFO] [multi] job_id 17071347 proc_lsst.multi INFO: [multi] job_id 17071347 2024-03-15 12:19:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-15 12:19:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071347 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071347 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/54b2f312-7960-494f-9ea0-ab6e4eece3f2_isrForCrosstalkSources_996706_58.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/54b2f312-7960-494f-9ea0-ab6e4eece3f2_isrForCrosstalkSources_996706_58.stderr parsl.executors.status_handling DEBUG: Launched block 156 on executor multi with job ID 17071347 parsl.executors.status_handling INFO: Allocated block ID 157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-157 2024-03-15 12:19:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 2024-03-15 12:19:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-15 12:19:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae963ebd54b16300170feb205e6b42fd with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/15471220-df71-436b-8439-b3048faaf34a_isrForCrosstalkSources_996710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/15471220-df71-436b-8439-b3048faaf34a_isrForCrosstalkSources_996710_9.stderr parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09342d1fc4b20d4b004c669abc773ebb with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f6c99c6b-f533-4aee-be1e-33505aa17724_isrForCrosstalkSources_996708_38.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f6c99c6b-f533-4aee-be1e-33505aa17724_isrForCrosstalkSources_996708_38.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c05e324fa48972e5a8f44a229b84306 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/d65d8366-293f-4c20-a4cb-25624f913d2b_isrForCrosstalkSources_996712_42.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/d65d8366-293f-4c20-a4cb-25624f913d2b_isrForCrosstalkSources_996712_42.stderr 2024-03-15 12:19:32 proc_lsst.multi:170 [INFO] [multi] job_id 17071348 proc_lsst.multi INFO: [multi] job_id 17071348 2024-03-15 12:19:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-15 12:19:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071348 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071348 parsl.executors.status_handling DEBUG: Launched block 157 on executor multi with job ID 17071348 parsl.executors.status_handling INFO: Allocated block ID 158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-158 2024-03-15 12:19:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9be8b5df68f109a6c5c54ac175cd5c with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/1a567327-a6e0-44c9-8d3c-192f1cf13075_isrForCrosstalkSources_996714_12.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/1a567327-a6e0-44c9-8d3c-192f1cf13075_isrForCrosstalkSources_996714_12.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4598b4acaeb491ec4a9ae0be4303f26 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/9ec7f57c-5f4a-4d60-9795-69f9d6c1e39b_isrForCrosstalkSources_996721_22.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/9ec7f57c-5f4a-4d60-9795-69f9d6c1e39b_isrForCrosstalkSources_996721_22.stderr parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a279f4d1062c7b000baf31292cb97555 with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/14117787-ca01-428f-980e-d68a4c16c22a_isrForCrosstalkSources_996710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/14117787-ca01-428f-980e-d68a4c16c22a_isrForCrosstalkSources_996710_60.stderr 2024-03-15 12:19:32 proc_lsst.multi:170 [INFO] [multi] job_id 17071349 proc_lsst.multi INFO: [multi] job_id 17071349 2024-03-15 12:19:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-15 12:19:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071349 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071349 parsl.executors.status_handling DEBUG: Launched block 158 on executor multi with job ID 17071349 parsl.executors.status_handling INFO: Allocated block ID 159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-159 2024-03-15 12:19:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-15 12:19:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4966b8b1b3fdd5fe9b43ae223f7ff9fc with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c6be1a3d-5259-4b7a-ba1c-a543e3c4e2c6_isrForCrosstalkSources_996705_51.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c6be1a3d-5259-4b7a-ba1c-a543e3c4e2c6_isrForCrosstalkSources_996705_51.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8652cba6bb0e0de459c733f23d598aed with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/2e560f20-9960-4b50-abbd-6abcdfdd342d_isrForCrosstalkSources_996707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/2e560f20-9960-4b50-abbd-6abcdfdd342d_isrForCrosstalkSources_996707_37.stderr parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a627661f3fc2fb3eb04f7549f5360ca with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/a42756f2-ac01-4383-933f-7507aeab5a69_isrForCrosstalkSources_996717_47.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/a42756f2-ac01-4383-933f-7507aeab5a69_isrForCrosstalkSources_996717_47.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdf681b74d5387f09008ad7a0d238169 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f07c29ba-061d-4082-aa19-619807450a5a_isrForCrosstalkSources_996709_31.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/f07c29ba-061d-4082-aa19-619807450a5a_isrForCrosstalkSources_996709_31.stderr parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9239e06376f8c7d38574d36bc257a953 with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/bcb18af3-b0e8-469f-8198-3d67a2ea299e_isrForCrosstalkSources_996709_9.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/bcb18af3-b0e8-469f-8198-3d67a2ea299e_isrForCrosstalkSources_996709_9.stderr parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 971828a69af23be3933e3354b66f3125 with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a4b23e60-7614-428e-b11a-6b0275944a6e_isrForCrosstalkSources_996708_13.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/a4b23e60-7614-428e-b11a-6b0275944a6e_isrForCrosstalkSources_996708_13.stderr 2024-03-15 12:19:33 proc_lsst.multi:170 [INFO] [multi] job_id 17071350 proc_lsst.multi INFO: [multi] job_id 17071350 2024-03-15 12:19:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-15 12:19:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071350 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071350 parsl.executors.status_handling DEBUG: Launched block 159 on executor multi with job ID 17071350 parsl.executors.status_handling INFO: Allocated block ID 160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-160 2024-03-15 12:19:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b938ed6c71311fdd75d969c43d70a9b3 with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/558161fc-853f-44fe-abcc-0979a3b146eb_isrForCrosstalkSources_996713_62.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/558161fc-853f-44fe-abcc-0979a3b146eb_isrForCrosstalkSources_996713_62.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 075fdd8600bc6b37719029d3be563e41 with result from task 884 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/82185aae-c485-4096-9d61-90520f77b1dd_isrForCrosstalkSources_996711_10.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/82185aae-c485-4096-9d61-90520f77b1dd_isrForCrosstalkSources_996711_10.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b064b6c05513e1e0d08e09d94af6718 with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dba5533c-5b3e-45cd-9dd4-3ef4e08f9009_isrForCrosstalkSources_996711_30.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dba5533c-5b3e-45cd-9dd4-3ef4e08f9009_isrForCrosstalkSources_996711_30.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acf36bb0b746fdeb6ecc337cf02b68c0 with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cee31caf-a3ef-401d-950f-e0dfde9eb5b8_isrForCrosstalkSources_996714_55.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cee31caf-a3ef-401d-950f-e0dfde9eb5b8_isrForCrosstalkSources_996714_55.stderr parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63f3c3ab9b3431623ed8c429442804a4 with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/a4d6c9a6-4d62-47cb-9f79-d1c4e6b2987f_isrForCrosstalkSources_996717_4.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/a4d6c9a6-4d62-47cb-9f79-d1c4e6b2987f_isrForCrosstalkSources_996717_4.stderr parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eb165b8285265cbe5175c1ed4be596d with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e68921bf-8395-4b91-8cc1-fd50a3f0d1eb_isrForCrosstalkSources_996720_1.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e68921bf-8395-4b91-8cc1-fd50a3f0d1eb_isrForCrosstalkSources_996720_1.stderr parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5a8e4736e9ac24d317293f8a9382a17 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e885c27c-1659-4b11-b6da-84f17ba5f61e_isrForCrosstalkSources_996721_31.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/e885c27c-1659-4b11-b6da-84f17ba5f61e_isrForCrosstalkSources_996721_31.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0954239ccffb92c406db20d57d25084 with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/56576489-5e98-4f60-9966-f2d521a90ba6_isrForCrosstalkSources_996712_62.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/56576489-5e98-4f60-9966-f2d521a90ba6_isrForCrosstalkSources_996712_62.stderr parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9240015e3d00343307c3033a149a0dcc with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/93dfcd11-7899-49c8-a35a-c42e904668e4_isrForCrosstalkSources_996721_26.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/93dfcd11-7899-49c8-a35a-c42e904668e4_isrForCrosstalkSources_996721_26.stderr parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 407161939cad377165a582e132fdc927 with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks 2024-03-15 12:19:33 proc_lsst.multi:170 [INFO] [multi] job_id 17071351 proc_lsst.multi INFO: [multi] job_id 17071351 2024-03-15 12:19:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-15 12:19:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071351 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071351 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a2287f53-508e-488d-9bdb-b5bedd897255_isrForCrosstalkSources_996719_57.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a2287f53-508e-488d-9bdb-b5bedd897255_isrForCrosstalkSources_996719_57.stderr parsl.executors.status_handling DEBUG: Launched block 160 on executor multi with job ID 17071351 parsl.executors.status_handling INFO: Allocated block ID 161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-161 2024-03-15 12:19:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-15 12:19:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16b5b4ab48b4a547d208b2808c8d8e6f with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f70a1644-19bc-4228-a4a8-f74a9a1cddc3_isrForCrosstalkSources_996713_26.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f70a1644-19bc-4228-a4a8-f74a9a1cddc3_isrForCrosstalkSources_996713_26.stderr parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3049a19b949ea072d6ca7bf478e468ef with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6a73b79d-123f-4860-ba77-ea631efa042f_isrForCrosstalkSources_996713_23.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6a73b79d-123f-4860-ba77-ea631efa042f_isrForCrosstalkSources_996713_23.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a09aec7138b352a3dc45dd035f41739a with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/2bf5aac3-9107-43d1-a72b-30652da090b3_isrForCrosstalkSources_996717_36.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/2bf5aac3-9107-43d1-a72b-30652da090b3_isrForCrosstalkSources_996717_36.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 286140843b05e94cb75924656c48d1ef with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5190a963-5d78-405c-b86c-c1e144dd8c51_isrForCrosstalkSources_996710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5190a963-5d78-405c-b86c-c1e144dd8c51_isrForCrosstalkSources_996710_4.stderr parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d358b749a8ae00d92bd823050e49dffb with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/05ae7d62-fa36-4b31-9fee-d0d7b6a035be_isrForCrosstalkSources_996720_31.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/05ae7d62-fa36-4b31-9fee-d0d7b6a035be_isrForCrosstalkSources_996720_31.stderr parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c9fe310fa805acec27c5f0916def5f5 with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8523ba52-3e5d-4784-9dbf-7f86e1aa5b2e_isrForCrosstalkSources_996718_7.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8523ba52-3e5d-4784-9dbf-7f86e1aa5b2e_isrForCrosstalkSources_996718_7.stderr parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f734eee24fdf5aa5edac2bade27b3755 with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cab1efc6-f687-414f-9cda-9cbdd7526bd7_isrForCrosstalkSources_996710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cab1efc6-f687-414f-9cda-9cbdd7526bd7_isrForCrosstalkSources_996710_14.stderr 2024-03-15 12:19:34 proc_lsst.multi:170 [INFO] [multi] job_id 17071352 proc_lsst.multi INFO: [multi] job_id 17071352 2024-03-15 12:19:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-15 12:19:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071352 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071352 parsl.executors.status_handling DEBUG: Launched block 161 on executor multi with job ID 17071352 parsl.executors.status_handling INFO: Allocated block ID 162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-162 2024-03-15 12:19:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 2024-03-15 12:19:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-15 12:19:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a8651cfcaef8ba642fc0a89e5d02d7a with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/bc25109c-c667-4f94-9ef6-2675536b2ab5_isrForCrosstalkSources_996706_45.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/bc25109c-c667-4f94-9ef6-2675536b2ab5_isrForCrosstalkSources_996706_45.stderr parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f13665699a0df09b62ed814ca986351 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/61b82d35-be8a-4af2-b10b-667cd46af7cc_isrForCrosstalkSources_996721_35.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/61b82d35-be8a-4af2-b10b-667cd46af7cc_isrForCrosstalkSources_996721_35.stderr parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c86c62f724aa1210110913d2eb8a5f3 with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/888f7e7b-3317-4dbc-b449-ef7b36683164_isrForCrosstalkSources_996714_9.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/888f7e7b-3317-4dbc-b449-ef7b36683164_isrForCrosstalkSources_996714_9.stderr parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81e99f8d55bc94b71c8f47dde22927f4 with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/35b72107-d3bb-4663-9fa1-0f11e2418c96_isrForCrosstalkSources_996706_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/35b72107-d3bb-4663-9fa1-0f11e2418c96_isrForCrosstalkSources_996706_26.stderr parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b7ebcad8f536113701eb069707efdc6 with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/dea16a6d-76f9-43d8-bf5c-0f8b7039606e_isrForCrosstalkSources_996710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/dea16a6d-76f9-43d8-bf5c-0f8b7039606e_isrForCrosstalkSources_996710_31.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64a8fd1eeab78677bd003a683f3d5716 with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/3e2aa1f8-a198-456b-b82d-4bd90e1e8424_isrForCrosstalkSources_996716_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/3e2aa1f8-a198-456b-b82d-4bd90e1e8424_isrForCrosstalkSources_996716_39.stderr parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d0d3a84cb530326f70fe0ad8caee10 with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/0f0cab85-3bb1-416f-b0ba-4ca251193c2f_isrForCrosstalkSources_996707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/0f0cab85-3bb1-416f-b0ba-4ca251193c2f_isrForCrosstalkSources_996707_5.stderr 2024-03-15 12:19:35 proc_lsst.multi:170 [INFO] [multi] job_id 17071353 proc_lsst.multi INFO: [multi] job_id 17071353 2024-03-15 12:19:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-15 12:19:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071353 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071353 parsl.executors.status_handling DEBUG: Launched block 162 on executor multi with job ID 17071353 parsl.executors.status_handling INFO: Allocated block ID 163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-163 2024-03-15 12:19:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 2024-03-15 12:19:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-15 12:19:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b169eed7a8a742b90af720ab8c22f6d5 with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c9ce87ec-6f6c-40f3-9e5c-42af1daac7cc_isrForCrosstalkSources_996712_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/c9ce87ec-6f6c-40f3-9e5c-42af1daac7cc_isrForCrosstalkSources_996712_28.stderr parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e79fb73c540aa364579ee5968568cf with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/32a20580-a695-4c5c-9c0e-cf1a5bf6d4d7_isrForCrosstalkSources_996707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/32a20580-a695-4c5c-9c0e-cf1a5bf6d4d7_isrForCrosstalkSources_996707_49.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2c5cc4df2632827d19932df5c07cc6d with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/77a175ae-1e9a-4e15-845a-f46b7e729ec6_isrForCrosstalkSources_996707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/77a175ae-1e9a-4e15-845a-f46b7e729ec6_isrForCrosstalkSources_996707_52.stderr parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26cec1e0eb276cd9d8d150e53db7c649 with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/aff884cc-ec89-4940-bc6f-6917358f657a_isrForCrosstalkSources_996717_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/aff884cc-ec89-4940-bc6f-6917358f657a_isrForCrosstalkSources_996717_21.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ec470fca66d39db289d97f4571d1b91 with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/99f0b31e-1aa3-4013-8ab4-2d522e50ceac_isrForCrosstalkSources_996717_3.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/99f0b31e-1aa3-4013-8ab4-2d522e50ceac_isrForCrosstalkSources_996717_3.stderr parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a987d1abfed4538a2e8e356f66f1752c with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/04e77813-2255-4a4c-9b3c-c7d91a23ad09_isrForCrosstalkSources_996709_4.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/04e77813-2255-4a4c-9b3c-c7d91a23ad09_isrForCrosstalkSources_996709_4.stderr parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8832f3aa9755614e9a3878d38ddede2a with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/fd4f27ec-5d9f-4811-8268-42871fd79685_isrForCrosstalkSources_996720_59.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/fd4f27ec-5d9f-4811-8268-42871fd79685_isrForCrosstalkSources_996720_59.stderr parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b712707a5208eeff7663dc3cd6a9db8a with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/98a3f2a7-1768-4823-b7e6-754752c29603_isrForCrosstalkSources_996716_28.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/98a3f2a7-1768-4823-b7e6-754752c29603_isrForCrosstalkSources_996716_28.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82655bced0821ecf682a47a75cc9cac7 with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/5fe78bef-05f5-4f40-9826-35833dff657b_isrForCrosstalkSources_996713_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/5fe78bef-05f5-4f40-9826-35833dff657b_isrForCrosstalkSources_996713_55.stderr parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 748a78fd2136868df80a2fb8b3a5c8a0 with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1fdfcedf-15be-4011-8322-93e3d439e70e_isrForCrosstalkSources_996712_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/1fdfcedf-15be-4011-8322-93e3d439e70e_isrForCrosstalkSources_996712_61.stderr 2024-03-15 12:19:36 proc_lsst.multi:170 [INFO] [multi] job_id 17071355 proc_lsst.multi INFO: [multi] job_id 17071355 2024-03-15 12:19:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-15 12:19:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071355 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071355 parsl.executors.status_handling DEBUG: Launched block 163 on executor multi with job ID 17071355 parsl.executors.status_handling INFO: Allocated block ID 164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-164 2024-03-15 12:19:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 2024-03-15 12:19:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-15 12:19:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28629fb72bc033779d149288a8aaa40e with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/1afa036c-8709-4407-824d-8e9649ad3f5b_isrForCrosstalkSources_996716_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/1afa036c-8709-4407-824d-8e9649ad3f5b_isrForCrosstalkSources_996716_3.stderr parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73ca0c8fc2453cb813c5c1114dc9be63 with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8683ead5-63a0-4757-95b3-24584fadec9e_isrForCrosstalkSources_996715_6.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/8683ead5-63a0-4757-95b3-24584fadec9e_isrForCrosstalkSources_996715_6.stderr parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3dcce300a7af4344267975770a2449b with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e50b19f3-1337-4d00-8d66-6d01af21269e_isrForCrosstalkSources_996709_43.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e50b19f3-1337-4d00-8d66-6d01af21269e_isrForCrosstalkSources_996709_43.stderr parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d25996650aff038de043d09b9d6c46ed with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/7a5ae8b1-f41f-4e98-a136-22812d10a9f1_isrForCrosstalkSources_996710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/7a5ae8b1-f41f-4e98-a136-22812d10a9f1_isrForCrosstalkSources_996710_62.stderr parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64540f7429b0b53cc48fc0b33541b116 with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/2c404240-1658-4806-9100-645388576e4e_isrForCrosstalkSources_996721_2.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/2c404240-1658-4806-9100-645388576e4e_isrForCrosstalkSources_996721_2.stderr parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca596e7014a543426b4416db78b8975c with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/64c2dc23-6eef-481d-8590-ad60cd1f45d8_isrForCrosstalkSources_996720_35.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/64c2dc23-6eef-481d-8590-ad60cd1f45d8_isrForCrosstalkSources_996720_35.stderr parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea5674efbee4d31038fe4f329c46002 with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/116e44f0-80b7-4c0d-aa87-c672624cee73_isrForCrosstalkSources_996706_25.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/116e44f0-80b7-4c0d-aa87-c672624cee73_isrForCrosstalkSources_996706_25.stderr 2024-03-15 12:19:37 proc_lsst.multi:170 [INFO] [multi] job_id 17071356 proc_lsst.multi INFO: [multi] job_id 17071356 2024-03-15 12:19:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-15 12:19:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071356 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071356 parsl.executors.status_handling DEBUG: Launched block 164 on executor multi with job ID 17071356 parsl.executors.status_handling INFO: Allocated block ID 165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-165 2024-03-15 12:19:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 2024-03-15 12:19:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-15 12:19:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6905e31d778f34f95effb64772192da3 with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8862ed49-21e5-4424-a5e4-bb8851511dd5_isrForCrosstalkSources_996717_8.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8862ed49-21e5-4424-a5e4-bb8851511dd5_isrForCrosstalkSources_996717_8.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5ae6039a898da3fb9a815b168f018fb with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/829e83bc-fa9e-49e6-bf3d-1da1f3b3bfa8_isrForCrosstalkSources_996721_13.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/829e83bc-fa9e-49e6-bf3d-1da1f3b3bfa8_isrForCrosstalkSources_996721_13.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3fe741bbc2425eabec5b354b4d56d92 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/c5574831-d42a-425c-a909-d928ad1bcbea_isrForCrosstalkSources_996716_51.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/c5574831-d42a-425c-a909-d928ad1bcbea_isrForCrosstalkSources_996716_51.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 115c4a6a1984862e9216d17f5019e320 with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/42d48cab-5627-4dec-962f-32f87b8d377d_isrForCrosstalkSources_996718_32.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/42d48cab-5627-4dec-962f-32f87b8d377d_isrForCrosstalkSources_996718_32.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6e4d0cc5ccfb00ca6ae1c36d062084a with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3fa0c897-5f4b-43ce-a551-1176a17ec879_isrForCrosstalkSources_996720_33.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3fa0c897-5f4b-43ce-a551-1176a17ec879_isrForCrosstalkSources_996720_33.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca2fd2d199fcd7a33052a539f8683ab5 with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/079515be-1b1c-4fd5-91c7-469cf774f8c6_isrForCrosstalkSources_996716_22.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/079515be-1b1c-4fd5-91c7-469cf774f8c6_isrForCrosstalkSources_996716_22.stderr parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f845ea772526b863f130db9a261fed9 with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1aa5e52e-fcaa-4dce-9606-ef0e9800787d_isrForCrosstalkSources_996707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/1aa5e52e-fcaa-4dce-9606-ef0e9800787d_isrForCrosstalkSources_996707_24.stderr parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49e2e85c488b224d8937206f2ca1f361 with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/53f23b29-5ea2-45e6-a849-aab3cc4897eb_isrForCrosstalkSources_996716_38.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/53f23b29-5ea2-45e6-a849-aab3cc4897eb_isrForCrosstalkSources_996716_38.stderr parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d85a54dc00d9f07258c94368883b740 with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0954d1e3-d773-43b4-8c47-0b377c3b8171_isrForCrosstalkSources_996719_19.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/0954d1e3-d773-43b4-8c47-0b377c3b8171_isrForCrosstalkSources_996719_19.stderr parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1df06f8a4ac93760d316851f3b6231e5 with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/bac5dfe4-764a-4ecc-816f-bf641b8d4849_isrForCrosstalkSources_996707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/bac5dfe4-764a-4ecc-816f-bf641b8d4849_isrForCrosstalkSources_996707_57.stderr parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ecfd3c949ac3fd01861ffce4a445045 with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b1f947d5-108b-4d4e-8524-3d7dc5093426_isrForCrosstalkSources_996712_41.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b1f947d5-108b-4d4e-8524-3d7dc5093426_isrForCrosstalkSources_996712_41.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77cab8224d09ee913f34ba8e8ff37920 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f1244c99-b13c-418c-b452-0f2b50f0b1f5_isrForCrosstalkSources_996712_19.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f1244c99-b13c-418c-b452-0f2b50f0b1f5_isrForCrosstalkSources_996712_19.stderr parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31914ca1c93cea56daee9bda5118971d with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/ed12b334-f885-49ea-8a2e-06bc2923df4a_isrForCrosstalkSources_996712_38.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/ed12b334-f885-49ea-8a2e-06bc2923df4a_isrForCrosstalkSources_996712_38.stderr 2024-03-15 12:19:38 proc_lsst.multi:170 [INFO] [multi] job_id 17071357 proc_lsst.multi INFO: [multi] job_id 17071357 2024-03-15 12:19:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-15 12:19:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071357 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071357 parsl.executors.status_handling DEBUG: Launched block 165 on executor multi with job ID 17071357 parsl.executors.status_handling INFO: Allocated block ID 166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-166 2024-03-15 12:19:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f2b18c0266e314dbc898fc26a53a5a6 with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/07ad7844-5d03-49da-9ff9-8d3147c7410b_isrForCrosstalkSources_996715_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/07ad7844-5d03-49da-9ff9-8d3147c7410b_isrForCrosstalkSources_996715_57.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d4f0fc374d5d6a84d35c951cbf10c60 with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/58676000-9240-405e-9eba-2bdcf70d8cd8_isrForCrosstalkSources_996710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/58676000-9240-405e-9eba-2bdcf70d8cd8_isrForCrosstalkSources_996710_32.stderr parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1214b207f3b66cbd17a47710fe341e7f with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/354c2e8d-5f15-4b92-a40c-676275d3fae1_isrForCrosstalkSources_996721_30.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/354c2e8d-5f15-4b92-a40c-676275d3fae1_isrForCrosstalkSources_996721_30.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03cc4ee93b9165cfc19c967f4241c8a3 with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f3459cb8-029c-40c4-9e3a-15f149752114_isrForCrosstalkSources_996705_18.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f3459cb8-029c-40c4-9e3a-15f149752114_isrForCrosstalkSources_996705_18.stderr 2024-03-15 12:19:38 proc_lsst.multi:170 [INFO] [multi] job_id 17071359 proc_lsst.multi INFO: [multi] job_id 17071359 2024-03-15 12:19:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-15 12:19:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071359 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071359 parsl.executors.status_handling DEBUG: Launched block 166 on executor multi with job ID 17071359 parsl.executors.status_handling INFO: Allocated block ID 167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-167 2024-03-15 12:19:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-15 12:19:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df7a118c14df43b0d0917d5373c9be15 with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/e539c47f-1af2-485d-9013-3e5c066a94bc_isrForCrosstalkSources_996714_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/e539c47f-1af2-485d-9013-3e5c066a94bc_isrForCrosstalkSources_996714_34.stderr parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 172a40b3e6b66832202209190023ab2b with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/ec8c41cf-99fe-4db8-a5d2-cfc56b5ea9e9_isrForCrosstalkSources_996719_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/ec8c41cf-99fe-4db8-a5d2-cfc56b5ea9e9_isrForCrosstalkSources_996719_21.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27459eb4580ebc691f4442c3c6b8fdd2 with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8df5573b-fb37-4fcf-9958-239e0624387e_isrForCrosstalkSources_996721_20.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8df5573b-fb37-4fcf-9958-239e0624387e_isrForCrosstalkSources_996721_20.stderr parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0daf8f4f428421f8f0eb96cce22fa7f with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b4208fca-f0e5-4499-bf2e-47380539325e_isrForCrosstalkSources_996712_33.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/b4208fca-f0e5-4499-bf2e-47380539325e_isrForCrosstalkSources_996712_33.stderr parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b729d1eb52387c7b4b1daf946b5e5ae with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5a3cb6e2-ddfd-46fa-9683-312c31777722_isrForCrosstalkSources_996709_16.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5a3cb6e2-ddfd-46fa-9683-312c31777722_isrForCrosstalkSources_996709_16.stderr parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c35410d0e6a2a4bdfa641d26f4a06310 with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5b2d3e85-f012-422a-a422-a4fea49c7dc3_isrForCrosstalkSources_996721_46.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/5b2d3e85-f012-422a-a422-a4fea49c7dc3_isrForCrosstalkSources_996721_46.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d43db0bbf4cb47d3a0dd2ed32529fea with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1fea4362-ffd1-43b6-abef-5fb1ae7b222c_isrForCrosstalkSources_996717_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1fea4362-ffd1-43b6-abef-5fb1ae7b222c_isrForCrosstalkSources_996717_33.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7458c5972c59905c356c8e1d3bfad674 with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/e74eeafc-d052-49ea-9b6b-61f94222c051_isrForCrosstalkSources_996710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/e74eeafc-d052-49ea-9b6b-61f94222c051_isrForCrosstalkSources_996710_30.stderr 2024-03-15 12:19:39 proc_lsst.multi:170 [INFO] [multi] job_id 17071360 proc_lsst.multi INFO: [multi] job_id 17071360 2024-03-15 12:19:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-15 12:19:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071360 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071360 parsl.executors.status_handling DEBUG: Launched block 167 on executor multi with job ID 17071360 parsl.executors.status_handling INFO: Allocated block ID 168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-168 2024-03-15 12:19:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 2024-03-15 12:19:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-15 12:19:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24bc6668da0e41d5b75fab086fe9de89 with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f46507b7-9fb4-4e22-9f37-7998faaa9e90_isrForCrosstalkSources_996711_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f46507b7-9fb4-4e22-9f37-7998faaa9e90_isrForCrosstalkSources_996711_38.stderr parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 713fd7e706c4262d0804c7e4a0079956 with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ac8e600b-922f-45fb-8e30-11fc157257d8_isrForCrosstalkSources_996716_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ac8e600b-922f-45fb-8e30-11fc157257d8_isrForCrosstalkSources_996716_30.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feec2a8f2d3e6a343f641d51c1d2116f with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1b3145b0-40ea-4aa6-8520-bb0b28c879d4_isrForCrosstalkSources_996721_16.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1b3145b0-40ea-4aa6-8520-bb0b28c879d4_isrForCrosstalkSources_996721_16.stderr parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c5dfe071e013b2badf2b16debc4de3e with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/13c3bc86-6abb-4d87-aa6a-babe84ccf314_isrForCrosstalkSources_996709_27.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/13c3bc86-6abb-4d87-aa6a-babe84ccf314_isrForCrosstalkSources_996709_27.stderr parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26bc863c5b881197ad66bc9b9c9042f9 with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/932c3d52-b883-46f3-9eff-1dbd2e22077f_isrForCrosstalkSources_996710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/932c3d52-b883-46f3-9eff-1dbd2e22077f_isrForCrosstalkSources_996710_42.stderr parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bd121250b2b0c331518d7147101d54c with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6a0ccf01-eb11-4439-a7a6-2ca76132355d_isrForCrosstalkSources_996711_7.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6a0ccf01-eb11-4439-a7a6-2ca76132355d_isrForCrosstalkSources_996711_7.stderr 2024-03-15 12:19:40 proc_lsst.multi:170 [INFO] [multi] job_id 17071361 proc_lsst.multi INFO: [multi] job_id 17071361 2024-03-15 12:19:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-15 12:19:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071361 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071361 parsl.executors.status_handling DEBUG: Launched block 168 on executor multi with job ID 17071361 parsl.executors.status_handling INFO: Allocated block ID 169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-169 2024-03-15 12:19:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 2024-03-15 12:19:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-15 12:19:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8a821f905ea8e989c8711b814e247f9 with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0566892c-25d4-4303-9078-4f2d15164f60_isrForCrosstalkSources_996720_39.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0566892c-25d4-4303-9078-4f2d15164f60_isrForCrosstalkSources_996720_39.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c570c09421ef7d871b586316f3bc704e with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/198b5a5a-9023-492b-adfa-79bd4733ddb6_isrForCrosstalkSources_996708_42.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/198b5a5a-9023-492b-adfa-79bd4733ddb6_isrForCrosstalkSources_996708_42.stderr parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc33e45f4c3c2053930403dfd4b3653c with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3ffc2d11-d11c-4987-85b2-439549a8c3af_isrForCrosstalkSources_996720_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/3ffc2d11-d11c-4987-85b2-439549a8c3af_isrForCrosstalkSources_996720_52.stderr parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d47e122d00c28b6927ed58c95dd5df00 with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/69deb6a3-2ad0-41b9-bfc0-31075ec1d936_isrForCrosstalkSources_996718_44.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/69deb6a3-2ad0-41b9-bfc0-31075ec1d936_isrForCrosstalkSources_996718_44.stderr parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7259b17622d43e944601ab63b0912922 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8ac1c605-635f-4630-9814-e47161dcdd9a_isrForCrosstalkSources_996717_9.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8ac1c605-635f-4630-9814-e47161dcdd9a_isrForCrosstalkSources_996717_9.stderr parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcb070133417e6e4b06a78f727c33456 with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/0b54b89d-17a2-4064-9500-2a572833c413_isrForCrosstalkSources_996717_10.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/0b54b89d-17a2-4064-9500-2a572833c413_isrForCrosstalkSources_996717_10.stderr parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0399c48f31fe6d69795fa16bd55e2722 with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3d0be745-cdf4-42ea-b5a9-23dd02572825_isrForCrosstalkSources_996711_43.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3d0be745-cdf4-42ea-b5a9-23dd02572825_isrForCrosstalkSources_996711_43.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2727283d5f63549649b6078a73cf7f4a with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/68ceb28b-f2dc-4e8a-b778-7feb937bf31a_isrForCrosstalkSources_996721_9.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/68ceb28b-f2dc-4e8a-b778-7feb937bf31a_isrForCrosstalkSources_996721_9.stderr parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04fff77aaba0df38734f791fd5538958 with result from task 992 parsl.dataflow.dflow DEBUG: Task 992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/40554e58-837e-42a4-a79e-33757dd2da4e_isrForCrosstalkSources_996717_27.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/40554e58-837e-42a4-a79e-33757dd2da4e_isrForCrosstalkSources_996717_27.stderr parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7520630ea0c8a0e362b5ea714c104768 with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d10d7f2a-083c-40de-9ffa-80e74fd6a0a8_isrForCrosstalkSources_996709_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d10d7f2a-083c-40de-9ffa-80e74fd6a0a8_isrForCrosstalkSources_996709_8.stderr parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4932d2bcc8bbfa2c70b29fb4919057e with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0c862462-4b11-4cd5-84ac-296eeba120fb_isrForCrosstalkSources_996720_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/0c862462-4b11-4cd5-84ac-296eeba120fb_isrForCrosstalkSources_996720_61.stderr parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79ef7c06de754fd1c162ab2b27bf0463 with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ea01dff7-4fa6-4aa4-82a1-ace6b1db09f6_isrForCrosstalkSources_996716_56.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/ea01dff7-4fa6-4aa4-82a1-ace6b1db09f6_isrForCrosstalkSources_996716_56.stderr parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276edd151af144af41220c9102709a73 with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fd297f4c-4188-46a4-8eb5-cd2521bf421d_isrForCrosstalkSources_996717_7.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fd297f4c-4188-46a4-8eb5-cd2521bf421d_isrForCrosstalkSources_996717_7.stderr parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f5fe7e27eb7e926ecdc4717310025c5 with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e30b2268-a3fc-454a-a626-0095cd6a6d61_isrForCrosstalkSources_996709_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e30b2268-a3fc-454a-a626-0095cd6a6d61_isrForCrosstalkSources_996709_24.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 220e7e8bd85ba93f420b8313727ab284 with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/87f3aa4e-5686-48da-a7fb-e16342600805_isrForCrosstalkSources_996709_48.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/87f3aa4e-5686-48da-a7fb-e16342600805_isrForCrosstalkSources_996709_48.stderr parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3487684413e0b166458d84359d94b9b with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/88079fe0-733c-47ac-bba6-e151dbf24a52_isrForCrosstalkSources_996714_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/88079fe0-733c-47ac-bba6-e151dbf24a52_isrForCrosstalkSources_996714_36.stderr parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77aa843c5ba62add68b5f0633d6ecac1 with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/14dcbf69-3e97-479d-8e72-986613f26a78_isrForCrosstalkSources_996710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/14dcbf69-3e97-479d-8e72-986613f26a78_isrForCrosstalkSources_996710_48.stderr parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d38b59f3729d3432a2d0e27a83018fd0 with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e8557416-925f-4740-850d-0118699e7536_isrForCrosstalkSources_996716_16.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/e8557416-925f-4740-850d-0118699e7536_isrForCrosstalkSources_996716_16.stderr parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6693ce8c45a39ac786db08753417868e with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/7d6e247a-d477-43c3-82f3-ccf5a85e699a_isrForCrosstalkSources_996717_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/7d6e247a-d477-43c3-82f3-ccf5a85e699a_isrForCrosstalkSources_996717_22.stderr 2024-03-15 12:19:41 proc_lsst.multi:170 [INFO] [multi] job_id 17071362 proc_lsst.multi INFO: [multi] job_id 17071362 2024-03-15 12:19:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-15 12:19:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071362 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071362 parsl.executors.status_handling DEBUG: Launched block 169 on executor multi with job ID 17071362 parsl.executors.status_handling INFO: Allocated block ID 170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-170 2024-03-15 12:19:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 2024-03-15 12:19:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-15 12:19:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fb539913036d11f8ee1ba1f797d717d with result from task 1019 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/cd29273a-3ccd-420b-8b51-d7de76c27727_isrForCrosstalkSources_996716_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/cd29273a-3ccd-420b-8b51-d7de76c27727_isrForCrosstalkSources_996716_55.stderr parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f6d8f082d441b0043588460a94ecfd7 with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4af084bb-d510-45aa-8300-9ed069370230_isrForCrosstalkSources_996718_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4af084bb-d510-45aa-8300-9ed069370230_isrForCrosstalkSources_996718_56.stderr parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dea961ec39534a43d1708e9f552e86da with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3b0df954-b1ff-4cc3-bc2e-b0c65c0bb961_isrForCrosstalkSources_996709_7.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/3b0df954-b1ff-4cc3-bc2e-b0c65c0bb961_isrForCrosstalkSources_996709_7.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7560f86d8b87b470efd5e6e1ed1bf5c9 with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/8eff4326-f3bc-40ad-bac8-36b41bb49264_isrForCrosstalkSources_996716_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/8eff4326-f3bc-40ad-bac8-36b41bb49264_isrForCrosstalkSources_996716_48.stderr parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67fc92716b8f03d19613f648b91f45e7 with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/631779bd-0c0d-4d6a-9896-3010caa4b487_isrForCrosstalkSources_996721_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/631779bd-0c0d-4d6a-9896-3010caa4b487_isrForCrosstalkSources_996721_15.stderr parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9083c271a32ae33645d867310407727b with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3c232d6d-1025-4bb6-9b45-bcc5a379c948_isrForCrosstalkSources_996705_43.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/3c232d6d-1025-4bb6-9b45-bcc5a379c948_isrForCrosstalkSources_996705_43.stderr parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f5fa5cfeb089833148d95a0ec42ab3a with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/52a4b8dd-1aa0-48ed-b8a7-cd2b2f8ad86d_isrForCrosstalkSources_996720_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/52a4b8dd-1aa0-48ed-b8a7-cd2b2f8ad86d_isrForCrosstalkSources_996720_51.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20e0bf8c7c97f7de0c329887fe390347 with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/678b4c40-3d08-4baf-bedc-2a1e0287aa30_isrForCrosstalkSources_996721_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/678b4c40-3d08-4baf-bedc-2a1e0287aa30_isrForCrosstalkSources_996721_33.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bf3f2edc2d11e0e89146cd46fc0bf8a with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/6bdc18a1-76f5-4965-add4-2b3a6ff3bbdb_isrForCrosstalkSources_996720_50.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/6bdc18a1-76f5-4965-add4-2b3a6ff3bbdb_isrForCrosstalkSources_996720_50.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce6de91c106a7b82ad909b18036506ba with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f627e300-18f8-40e0-8c43-44ce57302045_isrForCrosstalkSources_996711_19.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f627e300-18f8-40e0-8c43-44ce57302045_isrForCrosstalkSources_996711_19.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22969e9bfbdc4fa17ddd9a3608fc0cdb with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/b6a9d2f7-f0a0-441b-9276-8bee65044694_isrForCrosstalkSources_996717_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/b6a9d2f7-f0a0-441b-9276-8bee65044694_isrForCrosstalkSources_996717_20.stderr parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7682b430b6553b4fcd709ab819c2a4ea with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f52e5f0c-14a0-4f8c-9039-ea53ce9602fd_isrForCrosstalkSources_996705_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/f52e5f0c-14a0-4f8c-9039-ea53ce9602fd_isrForCrosstalkSources_996705_24.stderr parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d690acbfecc583adb4c29763f5ddf14 with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1d4f90a9-390a-47f7-ab23-340ffb20a7c5_isrForCrosstalkSources_996721_40.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1d4f90a9-390a-47f7-ab23-340ffb20a7c5_isrForCrosstalkSources_996721_40.stderr parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41ec6b25d5c086f4272ba581955ffd28 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4a3dc0fe-b902-4d85-9c17-be4bd72c0ae8_isrForCrosstalkSources_996713_10.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/4a3dc0fe-b902-4d85-9c17-be4bd72c0ae8_isrForCrosstalkSources_996713_10.stderr parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca952a59bb4cf2923482236242eeb29f with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/696f281e-e566-4b15-9fb1-38d8636d6709_isrForCrosstalkSources_996714_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/696f281e-e566-4b15-9fb1-38d8636d6709_isrForCrosstalkSources_996714_47.stderr parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24c05962ff1873fcfa80b43ee3af26e8 with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/74302b86-8b83-41ee-9a2e-e76d12162fcb_isrForCrosstalkSources_996708_50.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/74302b86-8b83-41ee-9a2e-e76d12162fcb_isrForCrosstalkSources_996708_50.stderr 2024-03-15 12:19:42 proc_lsst.multi:170 [INFO] [multi] job_id 17071363 proc_lsst.multi INFO: [multi] job_id 17071363 2024-03-15 12:19:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-15 12:19:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071363 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071363 parsl.executors.status_handling DEBUG: Launched block 170 on executor multi with job ID 17071363 parsl.executors.status_handling INFO: Allocated block ID 171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-171 2024-03-15 12:19:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 2024-03-15 12:19:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-15 12:19:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9974014c1e6c6c72afe7d315b49a9c97 with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/b423151c-766c-46e6-aad5-2f886377288d_isrForCrosstalkSources_996710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/b423151c-766c-46e6-aad5-2f886377288d_isrForCrosstalkSources_996710_21.stderr parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4312ae5ef3bc9bc3a719c3bbfe5cb41 with result from task 937 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e962f6ca-e683-4f8d-909b-e8190ad63a78_isrForCrosstalkSources_996713_48.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e962f6ca-e683-4f8d-909b-e8190ad63a78_isrForCrosstalkSources_996713_48.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ba5a8cb6afe285ca36f04e1f7dfaa7e with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/01cda1a7-7543-4c05-936e-16ad545ef790_isrForCrosstalkSources_996712_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/01cda1a7-7543-4c05-936e-16ad545ef790_isrForCrosstalkSources_996712_34.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b174dee5ee6c204bde98068b77422ed6 with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/b4c391bb-db3a-49e7-bc2d-a205494a7866_isrForCrosstalkSources_996708_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/b4c391bb-db3a-49e7-bc2d-a205494a7866_isrForCrosstalkSources_996708_43.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 415d0500cf12afd1b60cb552f1978f6a with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ce6a269b-ebd7-49c1-a65a-f6ca58f3391f_isrForCrosstalkSources_996713_30.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/ce6a269b-ebd7-49c1-a65a-f6ca58f3391f_isrForCrosstalkSources_996713_30.stderr parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bd6260b14b4f914a6d31da65f54b7ab with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/18b09049-0dab-4589-b168-4b0d93b3c331_isrForCrosstalkSources_996713_47.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/18b09049-0dab-4589-b168-4b0d93b3c331_isrForCrosstalkSources_996713_47.stderr parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 757bbd1c9520262fba2a4d2416b229c4 with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/756da7d0-2429-423c-9cc4-e7c1abc37d22_isrForCrosstalkSources_996719_61.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/756da7d0-2429-423c-9cc4-e7c1abc37d22_isrForCrosstalkSources_996719_61.stderr parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f12f28e66164f654b835ec5eba8710e2 with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4b06ea25-b112-4195-aa36-724e6105a337_isrForCrosstalkSources_996718_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/4b06ea25-b112-4195-aa36-724e6105a337_isrForCrosstalkSources_996718_52.stderr parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d12ed31994e79457194451968738c401 with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/31f8d234-d494-4d24-bdc5-354dac29a6be_isrForCrosstalkSources_996711_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/31f8d234-d494-4d24-bdc5-354dac29a6be_isrForCrosstalkSources_996711_52.stderr parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef277ec2f10e252758065631b7f3d13f with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b14dbc99-5d16-4e51-93b9-47d0e51bc4c5_isrForCrosstalkSources_996715_38.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b14dbc99-5d16-4e51-93b9-47d0e51bc4c5_isrForCrosstalkSources_996715_38.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8c2678827797890355d23981b2ad8ab with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e15787a7-aa73-4e8f-9358-974d090d434a_isrForCrosstalkSources_996717_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/e15787a7-aa73-4e8f-9358-974d090d434a_isrForCrosstalkSources_996717_55.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd0d98ef63873aecdb8fdd212a47572a with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b7516859-d1ac-4992-ab06-1e8de9a5f7ef_isrForCrosstalkSources_996706_19.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b7516859-d1ac-4992-ab06-1e8de9a5f7ef_isrForCrosstalkSources_996706_19.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffafa59ce3cafd0bd40b653dea8a0669 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e40dee66-074f-42e1-b1ff-24da304ab578_isrForCrosstalkSources_996713_46.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e40dee66-074f-42e1-b1ff-24da304ab578_isrForCrosstalkSources_996713_46.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 710c151605abde9e3c728f11a3f1e97e with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a4415684-fab1-4c0a-be71-6cec140fec2f_isrForCrosstalkSources_996706_14.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/a4415684-fab1-4c0a-be71-6cec140fec2f_isrForCrosstalkSources_996706_14.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9259336ef8ded9aa9c6c76eac8ea94cb with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/475a3c76-71d1-4cfd-9dbf-33be9b28df6d_isrForCrosstalkSources_996709_58.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/475a3c76-71d1-4cfd-9dbf-33be9b28df6d_isrForCrosstalkSources_996709_58.stderr parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4423a70b72e78cf0095471405e5765 with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/51868d7a-c92e-471a-b034-60c4167dd6b9_isrForCrosstalkSources_996708_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/51868d7a-c92e-471a-b034-60c4167dd6b9_isrForCrosstalkSources_996708_21.stderr parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cfb873c227e51e4315b5003812a343f with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f6f8f04f-7fe8-4b24-8d91-29dd0dc64e74_isrForCrosstalkSources_996717_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f6f8f04f-7fe8-4b24-8d91-29dd0dc64e74_isrForCrosstalkSources_996717_59.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25b64b613eceb0522e99595ac366dda8 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dce0fcb2-3d7f-4037-bc26-c31c71529acc_isrForCrosstalkSources_996711_24.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/dce0fcb2-3d7f-4037-bc26-c31c71529acc_isrForCrosstalkSources_996711_24.stderr parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdd74572ab2e1e86d0705378185aa964 with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/216ef912-bc2d-409b-af33-d57cc0761ae6_isrForCrosstalkSources_996717_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/216ef912-bc2d-409b-af33-d57cc0761ae6_isrForCrosstalkSources_996717_43.stderr 2024-03-15 12:19:43 proc_lsst.multi:170 [INFO] [multi] job_id 17071365 proc_lsst.multi INFO: [multi] job_id 17071365 2024-03-15 12:19:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-15 12:19:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071365 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071365 parsl.executors.status_handling DEBUG: Launched block 171 on executor multi with job ID 17071365 parsl.executors.status_handling INFO: Allocated block ID 172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-172 2024-03-15 12:19:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5382f047449b89665e78a3934853c1f3 with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f6447f4c-4b32-475e-aebd-16bfaa363499_isrForCrosstalkSources_996707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f6447f4c-4b32-475e-aebd-16bfaa363499_isrForCrosstalkSources_996707_36.stderr parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 453d74230d8ebf34fab40170b0a58793 with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/767b73ac-0d97-4b18-a0c3-e0be436496ea_isrForCrosstalkSources_996717_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/767b73ac-0d97-4b18-a0c3-e0be436496ea_isrForCrosstalkSources_996717_49.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c397912cb0cab7a0fa68976bbbe30dd2 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e451b33f-93c0-4118-a93e-0c89d8e603c0_isrForCrosstalkSources_996713_28.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/e451b33f-93c0-4118-a93e-0c89d8e603c0_isrForCrosstalkSources_996713_28.stderr 2024-03-15 12:19:43 proc_lsst.multi:170 [INFO] [multi] job_id 17071366 proc_lsst.multi INFO: [multi] job_id 17071366 2024-03-15 12:19:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-15 12:19:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071366 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071366 parsl.executors.status_handling DEBUG: Launched block 172 on executor multi with job ID 17071366 parsl.executors.status_handling INFO: Allocated block ID 173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-173 2024-03-15 12:19:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-15 12:19:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbe70a4bfb5ea4a198af67f6a6bf863f with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1d615cda-67da-4064-ae9f-5432c37fa18d_isrForCrosstalkSources_996705_31.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/1d615cda-67da-4064-ae9f-5432c37fa18d_isrForCrosstalkSources_996705_31.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3e0ba119db30d5118a04edcb3b84f5f with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3be8ff12-9d5d-4419-b671-3d993b23ebc9_isrForCrosstalkSources_996711_27.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3be8ff12-9d5d-4419-b671-3d993b23ebc9_isrForCrosstalkSources_996711_27.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6989ad076bba614f7d10fe37a9d703b6 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/5cbc30ba-0c4e-4683-8593-6efc4757e061_isrForCrosstalkSources_996716_44.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/5cbc30ba-0c4e-4683-8593-6efc4757e061_isrForCrosstalkSources_996716_44.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d40278a15f9b76553a7b2c875fbf4bf2 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/0655c96a-0153-46a9-9c06-823457d0174d_isrForCrosstalkSources_996710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/0655c96a-0153-46a9-9c06-823457d0174d_isrForCrosstalkSources_996710_58.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72b019c8737349e3a09f69b70db60a51 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1bf8001d-617c-4f1f-8dc5-16b6b91fe398_isrForCrosstalkSources_996717_48.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1bf8001d-617c-4f1f-8dc5-16b6b91fe398_isrForCrosstalkSources_996717_48.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 175be7804519a3c0b90fb4a909a309db with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3a7d183b-39e0-4428-bbdc-c86c5b929b45_isrForCrosstalkSources_996706_3.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/3a7d183b-39e0-4428-bbdc-c86c5b929b45_isrForCrosstalkSources_996706_3.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 531da7d74f1a064d93d2e82e2a1f7266 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f23d6ca6-91c1-4db0-96a2-ebcb55e716e1_isrForCrosstalkSources_996711_2.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f23d6ca6-91c1-4db0-96a2-ebcb55e716e1_isrForCrosstalkSources_996711_2.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fef7eec4d4ee594dd4b3b3807d511e24 with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/61ddc714-68c5-469f-9311-44191d673f8d_isrForCrosstalkSources_996716_7.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/61ddc714-68c5-469f-9311-44191d673f8d_isrForCrosstalkSources_996716_7.stderr parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45b1b8fc00c39303ec7c5fee586c66ef with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cae750c5-ce93-4ad0-be34-53d7920d3dea_isrForCrosstalkSources_996714_60.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cae750c5-ce93-4ad0-be34-53d7920d3dea_isrForCrosstalkSources_996714_60.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75bc5eac9792b3f75038344e88396cbd with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5d8ba5fd-0c1d-4d5d-9ba4-8d12e7df34a4_isrForCrosstalkSources_996709_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/5d8ba5fd-0c1d-4d5d-9ba4-8d12e7df34a4_isrForCrosstalkSources_996709_10.stderr parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27b503d43e64b8130f50a2958e757e2b with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9b4783e5-af54-46f2-aa55-19aac0585fe3_isrForCrosstalkSources_996714_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/9b4783e5-af54-46f2-aa55-19aac0585fe3_isrForCrosstalkSources_996714_6.stderr parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0857cba031a7997dd4ebbef1f1091fe9 with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/4dd49ead-241a-487f-b8d3-eb5d9259ab30_isrForCrosstalkSources_996708_62.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/4dd49ead-241a-487f-b8d3-eb5d9259ab30_isrForCrosstalkSources_996708_62.stderr parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2efe9809a2f8a5b907906b912152fea3 with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/80bc3169-27d3-4072-8c4d-af7416c21bf4_isrForCrosstalkSources_996715_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/80bc3169-27d3-4072-8c4d-af7416c21bf4_isrForCrosstalkSources_996715_23.stderr parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edb55dc2f9dbd6c9687451ad30957245 with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b94ba833-8ecd-42c1-9cea-827a2166711d_isrForCrosstalkSources_996706_60.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/b94ba833-8ecd-42c1-9cea-827a2166711d_isrForCrosstalkSources_996706_60.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a979f272e9f1509fc3efd7559de4ac22 with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/df3aea11-6a47-428c-bb02-53e6e4ab5b07_isrForCrosstalkSources_996711_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/df3aea11-6a47-428c-bb02-53e6e4ab5b07_isrForCrosstalkSources_996711_62.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 768f694c5ee8c02fade8f379ae6ebb21 with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/dca8013d-b107-4c1f-a1c6-5d24dbb3b5f0_isrForCrosstalkSources_996705_59.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/dca8013d-b107-4c1f-a1c6-5d24dbb3b5f0_isrForCrosstalkSources_996705_59.stderr parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c6dfe601fe7edab049df1ca69d6bdd with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c7f32692-bfb5-40f4-a611-1a7ea1743fdf_isrForCrosstalkSources_996709_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/c7f32692-bfb5-40f4-a611-1a7ea1743fdf_isrForCrosstalkSources_996709_50.stderr parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 615ea1b2d67dbbf19d7c07287515bff4 with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/060ac05c-18bd-4f5e-9a47-aa6a3c6dec2f_isrForCrosstalkSources_996705_54.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/060ac05c-18bd-4f5e-9a47-aa6a3c6dec2f_isrForCrosstalkSources_996705_54.stderr parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 761f4df54042825dc0253037eb20b222 with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c74f6e5f-50ae-4dcd-829f-7122751c3085_isrForCrosstalkSources_996706_44.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/c74f6e5f-50ae-4dcd-829f-7122751c3085_isrForCrosstalkSources_996706_44.stderr 2024-03-15 12:19:44 proc_lsst.multi:170 [INFO] [multi] job_id 17071367 proc_lsst.multi INFO: [multi] job_id 17071367 2024-03-15 12:19:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-15 12:19:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071367 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071367 parsl.executors.status_handling DEBUG: Launched block 173 on executor multi with job ID 17071367 parsl.executors.status_handling INFO: Allocated block ID 174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-174 2024-03-15 12:19:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 901172b6efcee080a48872366e58533f with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/81d5cd61-9e8f-4cf4-a754-98415608a768_isrForCrosstalkSources_996721_47.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/81d5cd61-9e8f-4cf4-a754-98415608a768_isrForCrosstalkSources_996721_47.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09ae6e37d88672c27edd0114968f65af with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5dff0b34-25f0-4bf4-bcad-50c70eb91770_isrForCrosstalkSources_996719_52.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5dff0b34-25f0-4bf4-bcad-50c70eb91770_isrForCrosstalkSources_996719_52.stderr parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a3868dd884bd6f0ee6d3fdb9631607b with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/83e88f25-13e8-4e9b-aa35-219ed4109a43_isrForCrosstalkSources_996712_37.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/83e88f25-13e8-4e9b-aa35-219ed4109a43_isrForCrosstalkSources_996712_37.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf7b713e3c9510de37ae8876f964525e with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/becc23f7-9a4f-474b-980e-8d12cfe8e22a_isrForCrosstalkSources_996711_41.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/becc23f7-9a4f-474b-980e-8d12cfe8e22a_isrForCrosstalkSources_996711_41.stderr parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9af8f9ead8f16bd094872775a87da74 with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2f30d21e-626b-46fd-99ee-5272dd14e736_isrForCrosstalkSources_996714_30.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/2f30d21e-626b-46fd-99ee-5272dd14e736_isrForCrosstalkSources_996714_30.stderr parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b7f0d0f705d21d71c43037fcd90416f with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/3291973a-2117-4962-8b80-769cb9398c0f_isrForCrosstalkSources_996718_42.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/3291973a-2117-4962-8b80-769cb9398c0f_isrForCrosstalkSources_996718_42.stderr parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e06fe90573fdffeb5dd931d3cfbcc27 with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/116066c0-0067-4023-b57e-409bdedd2dfb_isrForCrosstalkSources_996714_43.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/116066c0-0067-4023-b57e-409bdedd2dfb_isrForCrosstalkSources_996714_43.stderr parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86e3fef0aa04c464ee5b49cbe1928263 with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/78148eec-7ec6-47d5-8f1f-37a5adcb305f_isrForCrosstalkSources_996714_59.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/78148eec-7ec6-47d5-8f1f-37a5adcb305f_isrForCrosstalkSources_996714_59.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38fde6c752fd7152ed3a53a1007e327f with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/36857785-af52-4228-9da9-1e4c56cfc64d_isrForCrosstalkSources_996707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/36857785-af52-4228-9da9-1e4c56cfc64d_isrForCrosstalkSources_996707_42.stderr parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e708049def22517fc70d2c3c201f25 with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/796f2390-d6c6-4878-99df-4ed13a61f34f_isrForCrosstalkSources_996712_59.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/796f2390-d6c6-4878-99df-4ed13a61f34f_isrForCrosstalkSources_996712_59.stderr parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d73bded55a9289f8327ce715ea26a5f6 with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/0a78e455-8d60-4a9b-a776-054fcc2e7a92_isrForCrosstalkSources_996718_60.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/0a78e455-8d60-4a9b-a776-054fcc2e7a92_isrForCrosstalkSources_996718_60.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8cf6ed8b5778116f26719fa953774f2 with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/aa04b811-4aa6-4258-9657-339b30a54440_isrForCrosstalkSources_996719_55.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/aa04b811-4aa6-4258-9657-339b30a54440_isrForCrosstalkSources_996719_55.stderr 2024-03-15 12:19:44 proc_lsst.multi:170 [INFO] [multi] job_id 17071368 proc_lsst.multi INFO: [multi] job_id 17071368 2024-03-15 12:19:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-15 12:19:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071368 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071368 parsl.executors.status_handling DEBUG: Launched block 174 on executor multi with job ID 17071368 parsl.executors.status_handling INFO: Allocated block ID 175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-175 2024-03-15 12:19:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-15 12:19:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b462ec7c3d80ce3ac30f7b7e36d043ac with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3df6e00f-44e7-4c93-9069-6fdb877fb898_isrForCrosstalkSources_996719_45.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3df6e00f-44e7-4c93-9069-6fdb877fb898_isrForCrosstalkSources_996719_45.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9dcf33b13b8e70e6ede8294efce298d with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8829d55a-cfa9-498a-8e80-5c9a890bd8a7_isrForCrosstalkSources_996721_32.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/8829d55a-cfa9-498a-8e80-5c9a890bd8a7_isrForCrosstalkSources_996721_32.stderr parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91f0112d0b7dedd0d8f47c987c580c34 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fe6bd841-3667-4a2a-a0b2-391b37eb3b89_isrForCrosstalkSources_996717_1.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fe6bd841-3667-4a2a-a0b2-391b37eb3b89_isrForCrosstalkSources_996717_1.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a4d75d66fa3da07cd93c8b60965d97e with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/7c7e742a-65fb-4ab1-9782-d1a9c1e00f71_isrForCrosstalkSources_996713_40.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/7c7e742a-65fb-4ab1-9782-d1a9c1e00f71_isrForCrosstalkSources_996713_40.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bb69d59a1c240fce696b538b60e473d with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/af6abadf-b6a4-4a68-9b4b-5493d73b9149_isrForCrosstalkSources_996707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/af6abadf-b6a4-4a68-9b4b-5493d73b9149_isrForCrosstalkSources_996707_47.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 591d431927dbc944eaf61e8238d8979c with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/63798b98-8742-4e78-9c64-309e1196e94a_isrForCrosstalkSources_996706_8.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/63798b98-8742-4e78-9c64-309e1196e94a_isrForCrosstalkSources_996706_8.stderr parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f2955b33edd822d508db1224c2e45e6 with result from task 689 2024-03-15 12:19:45 proc_lsst.multi:170 [INFO] [multi] job_id 17071369 proc_lsst.multi INFO: [multi] job_id 17071369 2024-03-15 12:19:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-15 12:19:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071369 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071369 parsl.executors.status_handling DEBUG: Launched block 175 on executor multi with job ID 17071369 parsl.executors.status_handling INFO: Allocated block ID 176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-176 2024-03-15 12:19:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f4164667-3c89-40e0-8911-d47321716e1e_isrForCrosstalkSources_996716_27.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f4164667-3c89-40e0-8911-d47321716e1e_isrForCrosstalkSources_996716_27.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c604c7794a678fcc2c46e5ae9cc743a with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/48967121-50d2-460f-b70a-617677a88cd0_isrForCrosstalkSources_996706_28.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/48967121-50d2-460f-b70a-617677a88cd0_isrForCrosstalkSources_996706_28.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b49b052277f5b53179f84a7543e4feac with result from task 683 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/a8de82b9-e96d-4b87-ad9e-ed7ad1f2d0f2_isrForCrosstalkSources_996716_25.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/a8de82b9-e96d-4b87-ad9e-ed7ad1f2d0f2_isrForCrosstalkSources_996716_25.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8abe5d703bed0c9597c5de42d49ae70e with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/14f33a52-c732-4dc7-bcd3-799243443f17_isrForCrosstalkSources_996720_56.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/14f33a52-c732-4dc7-bcd3-799243443f17_isrForCrosstalkSources_996720_56.stderr parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d825ea46d2d842f319025385cd56157 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/de266c85-c632-475b-b6e5-6c901354e1bb_isrForCrosstalkSources_996721_39.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/de266c85-c632-475b-b6e5-6c901354e1bb_isrForCrosstalkSources_996721_39.stderr parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe61f894a43a8c0fcf550adb082a769 with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/b8244049-3cdb-421b-a52b-5f121edbdbc4_isrForCrosstalkSources_996707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/b8244049-3cdb-421b-a52b-5f121edbdbc4_isrForCrosstalkSources_996707_9.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ca0fb331828e7e1ff92b1d549818b51 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1867911f-08ec-4dfa-89c1-33e60a42a52a_isrForCrosstalkSources_996721_7.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/1867911f-08ec-4dfa-89c1-33e60a42a52a_isrForCrosstalkSources_996721_7.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d8027c17a6f88867e325a3e733addef with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/6ca7a0e5-1cd6-4678-8cd7-6802b757692f_isrForCrosstalkSources_996717_46.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/6ca7a0e5-1cd6-4678-8cd7-6802b757692f_isrForCrosstalkSources_996717_46.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82176d2d27818dab10b088f57b7dfefb with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5c899371-de50-461b-811b-689df1cb2c0f_isrForCrosstalkSources_996708_20.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/5c899371-de50-461b-811b-689df1cb2c0f_isrForCrosstalkSources_996708_20.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0f7c4eadd07dc6881fc4fef0182e1f8 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5062b486-c032-4adb-a528-952f7cdf5e1c_isrForCrosstalkSources_996710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5062b486-c032-4adb-a528-952f7cdf5e1c_isrForCrosstalkSources_996710_33.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8580e201be00e169c6644687ae1ff42 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/b5ff4568-658c-4df0-9901-d50f977f5558_isrForCrosstalkSources_996711_18.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/b5ff4568-658c-4df0-9901-d50f977f5558_isrForCrosstalkSources_996711_18.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46ad87e86ca8b503cf0f2bf78dd60ce4 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b458d01f-00c0-4bf7-a67f-32d57e6fb57a_isrForCrosstalkSources_996713_36.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/b458d01f-00c0-4bf7-a67f-32d57e6fb57a_isrForCrosstalkSources_996713_36.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b36b001061edcb2f283f3003f728b2b8 with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/fd5f43fa-0642-41e1-93a4-9aaa745b7e05_isrForCrosstalkSources_996713_9.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/fd5f43fa-0642-41e1-93a4-9aaa745b7e05_isrForCrosstalkSources_996713_9.stderr parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b08e4c8ad019f1c74ba31aeeffebb4a2 with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/d915f432-9b36-4ab6-88bc-b8942e44ff72_isrForCrosstalkSources_996710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/d915f432-9b36-4ab6-88bc-b8942e44ff72_isrForCrosstalkSources_996710_8.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a77553d33f7dee2d953d89b0f33c033 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ad1d3177-b410-46a3-afc0-6862010ae003_isrForCrosstalkSources_996707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/ad1d3177-b410-46a3-afc0-6862010ae003_isrForCrosstalkSources_996707_14.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee6b31fa4dd0992732badb48bf65296 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c7885d89-d166-402e-b0ec-84e77432abb0_isrForCrosstalkSources_996705_14.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/c7885d89-d166-402e-b0ec-84e77432abb0_isrForCrosstalkSources_996705_14.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3af3069828f4cd6d3aea4f8b889e20bd with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/62901d6d-2e73-4cec-8ed3-5b4e6552cd45_isrForCrosstalkSources_996711_48.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/62901d6d-2e73-4cec-8ed3-5b4e6552cd45_isrForCrosstalkSources_996711_48.stderr parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e685e157dd75c584ccb16f92af92e25d with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c1031d8f-7686-46a9-aa86-1a5ac6399f48_isrForCrosstalkSources_996714_14.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/c1031d8f-7686-46a9-aa86-1a5ac6399f48_isrForCrosstalkSources_996714_14.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30114f917b295a9aa8304e71380dbb84 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/bcc7ff04-a3d5-4b26-95c0-d49766386b6f_isrForCrosstalkSources_996706_33.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/bcc7ff04-a3d5-4b26-95c0-d49766386b6f_isrForCrosstalkSources_996706_33.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 554b976b762cb38f94e04436b9be5a28 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/5b2480cc-04ed-41c5-94b8-408438896e8e_isrForCrosstalkSources_996711_54.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/5b2480cc-04ed-41c5-94b8-408438896e8e_isrForCrosstalkSources_996711_54.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9432b4b55c386b6cc1b3aae4f21a4500 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/aa819c81-0e73-4860-9dee-b10f8a38f1cf_isrForCrosstalkSources_996716_32.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/aa819c81-0e73-4860-9dee-b10f8a38f1cf_isrForCrosstalkSources_996716_32.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7c685881fae457a2b812c3fc6a2d36e with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9c226a6f-0f2b-41d9-a8c6-32607cb7f617_isrForCrosstalkSources_996708_24.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/9c226a6f-0f2b-41d9-a8c6-32607cb7f617_isrForCrosstalkSources_996708_24.stderr parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8609eabca05af811bf54b9119e0acf0a with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/dd6fdd3a-1907-4a27-9967-9ec8cbbcade7_isrForCrosstalkSources_996715_52.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/dd6fdd3a-1907-4a27-9967-9ec8cbbcade7_isrForCrosstalkSources_996715_52.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41ff49c7961ed395bbde518ac52e6567 with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/56e0a820-a5d9-420d-913f-24b4b5078cab_isrForCrosstalkSources_996718_6.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/56e0a820-a5d9-420d-913f-24b4b5078cab_isrForCrosstalkSources_996718_6.stderr 2024-03-15 12:19:45 proc_lsst.multi:170 [INFO] [multi] job_id 17071370 proc_lsst.multi INFO: [multi] job_id 17071370 2024-03-15 12:19:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-15 12:19:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071370 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071370 parsl.executors.status_handling DEBUG: Launched block 176 on executor multi with job ID 17071370 parsl.executors.status_handling INFO: Allocated block ID 177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-177 2024-03-15 12:19:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44f0e2903e52c62f5cc7f37069ceec00 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7cb3a98a-4882-4a72-bf42-2e4b7f0d39a9_isrForCrosstalkSources_996715_12.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7cb3a98a-4882-4a72-bf42-2e4b7f0d39a9_isrForCrosstalkSources_996715_12.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6061ccd6e505c940136d41f1521894a2 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8c500436-d09d-4f48-9eed-1d06767452f5_isrForCrosstalkSources_996706_42.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8c500436-d09d-4f48-9eed-1d06767452f5_isrForCrosstalkSources_996706_42.stderr parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cac61e5b9ab766e0ec86f1e33a3c6564 with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/70fda952-aede-461e-b6d9-6d028eb3f69f_isrForCrosstalkSources_996712_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/70fda952-aede-461e-b6d9-6d028eb3f69f_isrForCrosstalkSources_996712_58.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57c13c7045e37a78035ef7e677b8c3d4 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7baa5323-6499-46c8-8aaa-1d9d6ade1eff_isrForCrosstalkSources_996715_44.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/7baa5323-6499-46c8-8aaa-1d9d6ade1eff_isrForCrosstalkSources_996715_44.stderr parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32042121560647c8e4940c6a479ce7bd with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/635666da-1d42-44a4-b325-fcd4a20d1a87_isrForCrosstalkSources_996717_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/635666da-1d42-44a4-b325-fcd4a20d1a87_isrForCrosstalkSources_996717_56.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cfcc3365822ced8ab2d0c90a4312a91 with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f7bbafe2-622d-452d-adb6-23ff734a5b19_isrForCrosstalkSources_996708_58.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/f7bbafe2-622d-452d-adb6-23ff734a5b19_isrForCrosstalkSources_996708_58.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80cd2121714d436dee5d5e660f03f663 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b6c7429f-2021-43a8-aee5-6aa03d6e47a0_isrForCrosstalkSources_996716_12.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/b6c7429f-2021-43a8-aee5-6aa03d6e47a0_isrForCrosstalkSources_996716_12.stderr parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a12ca0a3683d465c8493671ee8bf455 with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/4c1a671e-d8e7-4b95-9bbe-e45ddba3eb1b_isrForCrosstalkSources_996706_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/4c1a671e-d8e7-4b95-9bbe-e45ddba3eb1b_isrForCrosstalkSources_996706_22.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2588086f63604e631b37b07058509d85 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4a527693-5782-449e-abfb-e5b30c37482e_isrForCrosstalkSources_996707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/4a527693-5782-449e-abfb-e5b30c37482e_isrForCrosstalkSources_996707_11.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fb27ef2636a80bcaa7f68ae074fe189 with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e4ca6888-4904-49c5-93dd-33e4793bcede_isrForCrosstalkSources_996707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e4ca6888-4904-49c5-93dd-33e4793bcede_isrForCrosstalkSources_996707_56.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d2fd77e33f5c1e378dd4243301cff37 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cb893e40-c24c-4925-93e5-806961273bbd_isrForCrosstalkSources_996714_16.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/cb893e40-c24c-4925-93e5-806961273bbd_isrForCrosstalkSources_996714_16.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) 2024-03-15 12:19:45 proc_lsst.multi:170 [INFO] [multi] job_id 17071372 parsl.dataflow.memoization DEBUG: Storing app cache entry 3fab2d979a96ecf4a149d189a7685755 with result from task 659 proc_lsst.multi INFO: [multi] job_id 17071372 2024-03-15 12:19:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-15 12:19:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071372 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071372 parsl.executors.status_handling DEBUG: Launched block 177 on executor multi with job ID 17071372 parsl.executors.status_handling INFO: Allocated block ID 178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-178 2024-03-15 12:19:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow DEBUG: Task 659 checkpointed proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/7cb3d603-5a26-409f-b2f3-9e335b387804_isrForCrosstalkSources_996707_40.stdout proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/7cb3d603-5a26-409f-b2f3-9e335b387804_isrForCrosstalkSources_996707_40.stderr proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-15 12:19:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac977ec258cddcd4140887d29799bcef with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ddcb3d4b-5805-4306-89d1-27f179c324e9_isrForCrosstalkSources_996714_44.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ddcb3d4b-5805-4306-89d1-27f179c324e9_isrForCrosstalkSources_996714_44.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19087d215e747acc625b6f824122fe82 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/3a201cb0-38d5-46c1-84bf-bc66719c0750_isrForCrosstalkSources_996718_28.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/3a201cb0-38d5-46c1-84bf-bc66719c0750_isrForCrosstalkSources_996718_28.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fea93153fd568ed26ed7557cb6e5c9f9 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/96bb2524-e54d-4c3a-9d8b-f1b327f19ad4_isrForCrosstalkSources_996705_2.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/96bb2524-e54d-4c3a-9d8b-f1b327f19ad4_isrForCrosstalkSources_996705_2.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8b6927fe870d5b486b25ffaa4913f76 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5e10c774-8eee-448c-963e-ec79e4c308b0_isrForCrosstalkSources_996710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/5e10c774-8eee-448c-963e-ec79e4c308b0_isrForCrosstalkSources_996710_28.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca6e73f2808ab75c06f63c5fc4f70d83 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a82ae4d5-84ee-458b-8e08-0026df8c28c8_isrForCrosstalkSources_996719_41.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a82ae4d5-84ee-458b-8e08-0026df8c28c8_isrForCrosstalkSources_996719_41.stderr 2024-03-15 12:19:46 proc_lsst.multi:170 [INFO] [multi] job_id 17071373 proc_lsst.multi INFO: [multi] job_id 17071373 2024-03-15 12:19:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-15 12:19:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071373 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071373 parsl.executors.status_handling DEBUG: Launched block 178 on executor multi with job ID 17071373 parsl.executors.status_handling INFO: Allocated block ID 179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-179 2024-03-15 12:19:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abfd6a88b9eb91e8715128354d740683 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f315a9db-8395-476c-a734-8b5669168960_isrForCrosstalkSources_996713_3.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/f315a9db-8395-476c-a734-8b5669168960_isrForCrosstalkSources_996713_3.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2881c8bb42d822588152d6891deb85b2 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/a7057946-401a-44a5-a28f-ff139ad50fd6_isrForCrosstalkSources_996711_16.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/a7057946-401a-44a5-a28f-ff139ad50fd6_isrForCrosstalkSources_996711_16.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e1b685abf88dfef0ec7d5adc4d99f35 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e86b85e6-6595-4b2d-b9d6-a594a794436c_isrForCrosstalkSources_996709_62.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/e86b85e6-6595-4b2d-b9d6-a594a794436c_isrForCrosstalkSources_996709_62.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 183390c7485fefb1bad92412731fb9fb with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/261b534e-080e-4857-9595-f51ada08b25e_isrForCrosstalkSources_996710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/261b534e-080e-4857-9595-f51ada08b25e_isrForCrosstalkSources_996710_61.stderr 2024-03-15 12:19:46 proc_lsst.multi:170 [INFO] [multi] job_id 17071374 proc_lsst.multi INFO: [multi] job_id 17071374 2024-03-15 12:19:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-15 12:19:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071374 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071374 parsl.executors.status_handling DEBUG: Launched block 179 on executor multi with job ID 17071374 parsl.executors.status_handling INFO: Allocated block ID 180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-180 2024-03-15 12:19:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-15 12:19:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d53236c89e1509024310746e041459ec with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/22423cd8-57de-4dda-899e-74e5514f468c_isrForCrosstalkSources_996716_52.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/22423cd8-57de-4dda-899e-74e5514f468c_isrForCrosstalkSources_996716_52.stderr parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d0fd0b10b11ae1e0dd6e6405af851dc with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a76c5c09-7189-4d05-8052-0c592a6ab786_isrForCrosstalkSources_996713_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a76c5c09-7189-4d05-8052-0c592a6ab786_isrForCrosstalkSources_996713_57.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93eeb7544909e04a0c358594b7829aa4 with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/8ccbf1a9-f72b-41b3-b98d-c42519948f78_isrForCrosstalkSources_996719_54.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/8ccbf1a9-f72b-41b3-b98d-c42519948f78_isrForCrosstalkSources_996719_54.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbb3ec7e883284375ffeb567348a5c1 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/3bb66f9e-bda1-41a3-8afd-609613acc80a_isrForCrosstalkSources_996721_41.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/3bb66f9e-bda1-41a3-8afd-609613acc80a_isrForCrosstalkSources_996721_41.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 880e8f6422ea4ed8fea599f4100ac50f with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a2e874af-caa1-48e7-b9cd-5bf59811cddb_isrForCrosstalkSources_996713_41.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/a2e874af-caa1-48e7-b9cd-5bf59811cddb_isrForCrosstalkSources_996713_41.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d23b4605c87eef77d706c3b5676b463a with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/02bf03cd-9eec-409c-a3e8-fa333983b847_isrForCrosstalkSources_996721_3.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/02bf03cd-9eec-409c-a3e8-fa333983b847_isrForCrosstalkSources_996721_3.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e468d796f2be67777d7f7680905d696 with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/2ba1b89d-f171-4627-b18e-580e79c163e3_isrForCrosstalkSources_996721_5.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/2ba1b89d-f171-4627-b18e-580e79c163e3_isrForCrosstalkSources_996721_5.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c270b862db2c8e8bd49fc7308fec82 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c08724f0-c5c5-4dd5-9446-e464767cf6e1_isrForCrosstalkSources_996719_36.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/c08724f0-c5c5-4dd5-9446-e464767cf6e1_isrForCrosstalkSources_996719_36.stderr parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 727cdad7b2457d339bda68888a51f989 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/9c404c15-4c68-4893-980c-6dc930732078_isrForCrosstalkSources_996711_42.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/9c404c15-4c68-4893-980c-6dc930732078_isrForCrosstalkSources_996711_42.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 305a58da318975e27691c9b4e6a245b8 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/7d914812-ab8f-4e52-99e6-8b1cb19d8309_isrForCrosstalkSources_996708_25.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/7d914812-ab8f-4e52-99e6-8b1cb19d8309_isrForCrosstalkSources_996708_25.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0b04165621e0f5507a551ec327ea9bf with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/12b62d9b-64f8-4d71-96ee-c794e00e1226_isrForCrosstalkSources_996712_4.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/12b62d9b-64f8-4d71-96ee-c794e00e1226_isrForCrosstalkSources_996712_4.stderr parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7758c14761caffcc18ae032d574bac8b with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/37e8ca3e-bfae-4e5e-946a-2f76ea5f2abd_isrForCrosstalkSources_996708_26.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/37e8ca3e-bfae-4e5e-946a-2f76ea5f2abd_isrForCrosstalkSources_996708_26.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23390c506877e5faae489816e15221ee with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ecfb1e7f-302d-4023-acd5-5c4061a11c21_isrForCrosstalkSources_996706_59.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ecfb1e7f-302d-4023-acd5-5c4061a11c21_isrForCrosstalkSources_996706_59.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 281022bc9dd72f9b24fe95e24345b613 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6883b5b3-52e5-4389-849b-7b5bcb35990c_isrForCrosstalkSources_996710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/6883b5b3-52e5-4389-849b-7b5bcb35990c_isrForCrosstalkSources_996710_52.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69e311c9613e3dee0378e29a8faf5f11 with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ac29fd84-4a75-4d90-a232-501dd1cfa8c5_isrForCrosstalkSources_996706_46.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/ac29fd84-4a75-4d90-a232-501dd1cfa8c5_isrForCrosstalkSources_996706_46.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd9efc1edfe6e55ad43c42c2f6d2b5aa with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/408f693c-187c-4631-b3a4-2ea643d733d3_isrForCrosstalkSources_996708_6.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/408f693c-187c-4631-b3a4-2ea643d733d3_isrForCrosstalkSources_996708_6.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f07a633ab9c0bd0276e8c3a1e47fbab with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/bed6980b-1674-4e5e-b9d3-ceb023555256_isrForCrosstalkSources_996707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/bed6980b-1674-4e5e-b9d3-ceb023555256_isrForCrosstalkSources_996707_22.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a9451483c211fec436e51fa5f51f4ad with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/4a505199-cb02-40a7-b14c-d44a1747b022_isrForCrosstalkSources_996710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/4a505199-cb02-40a7-b14c-d44a1747b022_isrForCrosstalkSources_996710_36.stderr parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aca3c8ded18db2bd3eab99e0f893725f with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/d5bc9797-a566-45a5-9275-c5b767dbe80d_isrForCrosstalkSources_996718_23.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/d5bc9797-a566-45a5-9275-c5b767dbe80d_isrForCrosstalkSources_996718_23.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2948a29c6b5a18112fe0151468ae6091 with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/ca7ebcc0-8aba-4b03-9148-f1f249d74d2b_isrForCrosstalkSources_996715_13.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/ca7ebcc0-8aba-4b03-9148-f1f249d74d2b_isrForCrosstalkSources_996715_13.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a20c12bf107a71f7d7b58ceb5de47072 with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f2ae9040-bccb-4cff-9095-8bdb43ec659a_isrForCrosstalkSources_996707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/f2ae9040-bccb-4cff-9095-8bdb43ec659a_isrForCrosstalkSources_996707_51.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6157c85afeb873d7fae3d8d08b531c3 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/66025669-b784-4b4a-877e-d75cbd64e36e_isrForCrosstalkSources_996719_53.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/66025669-b784-4b4a-877e-d75cbd64e36e_isrForCrosstalkSources_996719_53.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0d256d5f908452a6bee5b192728f9a3 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/a1e39d73-7cf2-4499-ae4f-98faa14790e7_isrForCrosstalkSources_996710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/a1e39d73-7cf2-4499-ae4f-98faa14790e7_isrForCrosstalkSources_996710_17.stderr parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc99f9613f305a07658f1e1383ed69c2 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/9dfbe05f-950d-448e-8ece-c14fc761213c_isrForCrosstalkSources_996716_14.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/9dfbe05f-950d-448e-8ece-c14fc761213c_isrForCrosstalkSources_996716_14.stderr parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd4bc17e4e964878df29ecd7d1badfe3 with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/683f0112-4abd-4eeb-a265-43becdeef65d_isrForCrosstalkSources_996708_7.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996708/683f0112-4abd-4eeb-a265-43becdeef65d_isrForCrosstalkSources_996708_7.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78110e3ff334c58e314d694c3934827f with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/963a0b81-41dc-45e2-9a2a-b46256dbe6a0_isrForCrosstalkSources_996720_54.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/963a0b81-41dc-45e2-9a2a-b46256dbe6a0_isrForCrosstalkSources_996720_54.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37b899e0ed31fe57d825e570a9060605 with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/d7cd9c3b-7309-44c6-9dda-e7cdc7be2f6d_isrForCrosstalkSources_996720_45.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/d7cd9c3b-7309-44c6-9dda-e7cdc7be2f6d_isrForCrosstalkSources_996720_45.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7572c87f0ded1c645bcc4512d31329 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a87110db-ea0d-4798-b25a-83f220cf50e7_isrForCrosstalkSources_996720_8.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/a87110db-ea0d-4798-b25a-83f220cf50e7_isrForCrosstalkSources_996720_8.stderr 2024-03-15 12:19:47 proc_lsst.multi:170 [INFO] [multi] job_id 17071376 proc_lsst.multi INFO: [multi] job_id 17071376 2024-03-15 12:19:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-15 12:19:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071376 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071376 parsl.executors.status_handling DEBUG: Launched block 180 on executor multi with job ID 17071376 parsl.executors.status_handling INFO: Allocated block ID 181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-181 2024-03-15 12:19:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7c0e25ca0a2714506c75ffd04001ff3 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/608af338-da1e-45e5-bc6e-3993ce2675d5_isrForCrosstalkSources_996706_61.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/608af338-da1e-45e5-bc6e-3993ce2675d5_isrForCrosstalkSources_996706_61.stderr parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d84ecefce40af8120016987d109085d0 with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8ab194dd-3731-4ef9-bd50-d81e4325b7ed_isrForCrosstalkSources_996718_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/8ab194dd-3731-4ef9-bd50-d81e4325b7ed_isrForCrosstalkSources_996718_29.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a5f0383328faa1f7fb2c3fc33109f4c with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/c69250c3-7777-44b0-b28b-700d5320e007_isrForCrosstalkSources_996710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/c69250c3-7777-44b0-b28b-700d5320e007_isrForCrosstalkSources_996710_29.stderr 2024-03-15 12:19:47 proc_lsst.multi:170 [INFO] [multi] job_id 17071377 proc_lsst.multi INFO: [multi] job_id 17071377 2024-03-15 12:19:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-15 12:19:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071377 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071377 parsl.executors.status_handling DEBUG: Launched block 181 on executor multi with job ID 17071377 parsl.executors.status_handling INFO: Allocated block ID 182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-182 2024-03-15 12:19:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff360b10907dac5a8ae5e8a6d28ea773 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/2bd8210c-4601-4d78-9e4b-2cf39b68f954_isrForCrosstalkSources_996715_55.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/2bd8210c-4601-4d78-9e4b-2cf39b68f954_isrForCrosstalkSources_996715_55.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38a9a1c91d9600071085a8fdc14d8f3f with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/aeb11c8d-8c99-4bdb-8a05-59f628ba9621_isrForCrosstalkSources_996707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/aeb11c8d-8c99-4bdb-8a05-59f628ba9621_isrForCrosstalkSources_996707_50.stderr parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f44982a670e32a82547110aea817662a with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4b9e0e84-8ece-4be5-a11d-c62c237db681_isrForCrosstalkSources_996714_33.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/4b9e0e84-8ece-4be5-a11d-c62c237db681_isrForCrosstalkSources_996714_33.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a050c98d97a1634f465cba25aa6c1d3 with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f66d1cd7-63c2-45ff-bf9b-8e39801cc587_isrForCrosstalkSources_996716_26.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/f66d1cd7-63c2-45ff-bf9b-8e39801cc587_isrForCrosstalkSources_996716_26.stderr 2024-03-15 12:19:47 proc_lsst.multi:170 [INFO] [multi] job_id 17071378 proc_lsst.multi INFO: [multi] job_id 17071378 2024-03-15 12:19:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-15 12:19:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071378 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071378 parsl.executors.status_handling DEBUG: Launched block 182 on executor multi with job ID 17071378 parsl.executors.status_handling INFO: Allocated block ID 183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-183 2024-03-15 12:19:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-15 12:19:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:48 proc_lsst.multi:170 [INFO] [multi] job_id 17071379 proc_lsst.multi INFO: [multi] job_id 17071379 2024-03-15 12:19:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-15 12:19:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071379 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071379 parsl.executors.status_handling DEBUG: Launched block 183 on executor multi with job ID 17071379 parsl.executors.status_handling INFO: Allocated block ID 184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-184 2024-03-15 12:19:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:48 proc_lsst.multi:170 [INFO] [multi] job_id 17071380 proc_lsst.multi INFO: [multi] job_id 17071380 2024-03-15 12:19:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-15 12:19:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071380 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071380 parsl.executors.status_handling DEBUG: Launched block 184 on executor multi with job ID 17071380 parsl.executors.status_handling INFO: Allocated block ID 185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-185 2024-03-15 12:19:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-15 12:19:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:49 proc_lsst.multi:170 [INFO] [multi] job_id 17071381 proc_lsst.multi INFO: [multi] job_id 17071381 2024-03-15 12:19:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-15 12:19:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071381 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071381 parsl.executors.status_handling DEBUG: Launched block 185 on executor multi with job ID 17071381 parsl.executors.status_handling INFO: Allocated block ID 186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-186 2024-03-15 12:19:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a07e5ff933411aa42961ce1cd8aad14d with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/465ae9b2-5b6f-4e4d-8562-557ad76c8b26_isrForCrosstalkSources_996713_27.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/465ae9b2-5b6f-4e4d-8562-557ad76c8b26_isrForCrosstalkSources_996713_27.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d64c393dcf53e413055746c94ba6a765 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/d8bdc16c-e3ca-490d-94f5-18520e0d7cff_isrForCrosstalkSources_996707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/d8bdc16c-e3ca-490d-94f5-18520e0d7cff_isrForCrosstalkSources_996707_26.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db6b7b56a7cdef0ba3e988b38cd8b7a1 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/82c69b8c-008d-4daf-9e66-1b2c89aa1808_isrForCrosstalkSources_996706_39.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/82c69b8c-008d-4daf-9e66-1b2c89aa1808_isrForCrosstalkSources_996706_39.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8987ad4bcd3d6d01751a6eac3eb88f51 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/0983fa0c-cba3-4468-86af-520d9703c258_isrForCrosstalkSources_996707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/0983fa0c-cba3-4468-86af-520d9703c258_isrForCrosstalkSources_996707_23.stderr parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 239942ba5006095fef36f6423ac4347b with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/b814e9f7-1723-4e07-9fdd-c8599b4f3d61_isrForCrosstalkSources_996709_39.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/b814e9f7-1723-4e07-9fdd-c8599b4f3d61_isrForCrosstalkSources_996709_39.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3c2e10e0324b423a8d4a7241b516b09 with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c6212a84-4a84-43fa-962b-aa6137468dc0_isrForCrosstalkSources_996713_34.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/c6212a84-4a84-43fa-962b-aa6137468dc0_isrForCrosstalkSources_996713_34.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8430ca809fe698f5e830c77389920905 with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a4da0604-06b8-418d-bbe6-f5331f885138_isrForCrosstalkSources_996719_12.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/a4da0604-06b8-418d-bbe6-f5331f885138_isrForCrosstalkSources_996719_12.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d9777f132e65ddb0fafc785110630d2 with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3dc8824f-db04-44a7-9ca0-aaf6ee0d207d_isrForCrosstalkSources_996711_14.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/3dc8824f-db04-44a7-9ca0-aaf6ee0d207d_isrForCrosstalkSources_996711_14.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b359aa77178d7db71163e8f67e0263c2 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/81f7a664-81e4-4921-9835-e7b66c71a4a0_isrForCrosstalkSources_996717_32.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/81f7a664-81e4-4921-9835-e7b66c71a4a0_isrForCrosstalkSources_996717_32.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a675ac660debc7907c8bf33c6a595eb with result from task 745 2024-03-15 12:19:49 proc_lsst.multi:170 [INFO] [multi] job_id 17071382 proc_lsst.multi INFO: [multi] job_id 17071382 2024-03-15 12:19:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-15 12:19:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071382 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071382 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.executors.status_handling DEBUG: Launched block 186 on executor multi with job ID 17071382 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.executors.status_handling INFO: Allocated block ID 187 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/3d050409-6b2c-4468-bb66-7867b40e9a67_isrForCrosstalkSources_996717_61.stdout parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-187 2024-03-15 12:19:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/3d050409-6b2c-4468-bb66-7867b40e9a67_isrForCrosstalkSources_996717_61.stderr proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-15 12:19:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e38be7df33b6f614a686c5a85cc2096 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ae220146-660e-4852-9e8c-64b98d0f6b06_isrForCrosstalkSources_996709_36.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/ae220146-660e-4852-9e8c-64b98d0f6b06_isrForCrosstalkSources_996709_36.stderr parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0bf744c40b0f9cfc972667ce1f30318 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/4b3924e4-0c7e-4980-b380-05e9b08e598d_isrForCrosstalkSources_996705_36.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/4b3924e4-0c7e-4980-b380-05e9b08e598d_isrForCrosstalkSources_996705_36.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3871454b2145abb343a708cd553ad58e with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e82ba63a-3ddb-469c-a98e-b5404d94987c_isrForCrosstalkSources_996707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/e82ba63a-3ddb-469c-a98e-b5404d94987c_isrForCrosstalkSources_996707_41.stderr 2024-03-15 12:19:50 proc_lsst.multi:170 [INFO] [multi] job_id 17071383 proc_lsst.multi INFO: [multi] job_id 17071383 2024-03-15 12:19:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-15 12:19:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071383 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071383 parsl.executors.status_handling DEBUG: Launched block 187 on executor multi with job ID 17071383 parsl.executors.status_handling INFO: Allocated block ID 188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-188 2024-03-15 12:19:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 775252fb2e62b475ac34c3ed741ab3f9 with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b865dd2c-f1b6-467c-904a-cad32cdba1b5_isrForCrosstalkSources_996715_50.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/b865dd2c-f1b6-467c-904a-cad32cdba1b5_isrForCrosstalkSources_996715_50.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cef88d5a88abd42400912cd8a35bebfb with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f05f2f59-8b02-430a-b40e-10969107d510_isrForCrosstalkSources_996712_35.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/f05f2f59-8b02-430a-b40e-10969107d510_isrForCrosstalkSources_996712_35.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 058f8a1d4c63cb1870d9c63ab500dac5 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/9d9488f6-fdc5-4541-a56d-aa58fa6b0b67_isrForCrosstalkSources_996712_44.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/9d9488f6-fdc5-4541-a56d-aa58fa6b0b67_isrForCrosstalkSources_996712_44.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c111a0d7b195accb5a19aff01e502735 with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/b4b50147-311e-42cd-9f3b-4d14f6f13f63_isrForCrosstalkSources_996714_13.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/b4b50147-311e-42cd-9f3b-4d14f6f13f63_isrForCrosstalkSources_996714_13.stderr parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3f9e9eac0d14e32f0118f19f9c5c584 with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/3e2552cc-9f33-4cff-9e14-eb2f8bd975ea_isrForCrosstalkSources_996712_5.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/3e2552cc-9f33-4cff-9e14-eb2f8bd975ea_isrForCrosstalkSources_996712_5.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d307234de52b2ae39c2461f9d348192e with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ca30dd63-d485-4256-9d6a-1ffe7f09130d_isrForCrosstalkSources_996718_26.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/ca30dd63-d485-4256-9d6a-1ffe7f09130d_isrForCrosstalkSources_996718_26.stderr parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9982fb18021d241afc7db43bcef0a7ab with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/baf53551-ea9b-4a09-bffd-36271e11e577_isrForCrosstalkSources_996710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/baf53551-ea9b-4a09-bffd-36271e11e577_isrForCrosstalkSources_996710_37.stderr parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4940a2f75be22427448d151a12df9c75 with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3b38afba-8498-4684-b855-01f1a6d5d9c0_isrForCrosstalkSources_996719_4.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/3b38afba-8498-4684-b855-01f1a6d5d9c0_isrForCrosstalkSources_996719_4.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7627b0870ac0e22391ae44824ac946c7 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/b718d40b-087c-4216-b1aa-fd8aa0e51d18_isrForCrosstalkSources_996717_12.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/b718d40b-087c-4216-b1aa-fd8aa0e51d18_isrForCrosstalkSources_996717_12.stderr 2024-03-15 12:19:50 proc_lsst.multi:170 [INFO] [multi] job_id 17071384 proc_lsst.multi INFO: [multi] job_id 17071384 2024-03-15 12:19:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-15 12:19:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071384 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071384 parsl.executors.status_handling DEBUG: Launched block 188 on executor multi with job ID 17071384 parsl.executors.status_handling INFO: Allocated block ID 189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-189 2024-03-15 12:19:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ecfd03f463376522ac6f9065d4ef49e with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/5bb9150c-87d4-47a5-999e-5304d45c3867_isrForCrosstalkSources_996715_31.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/5bb9150c-87d4-47a5-999e-5304d45c3867_isrForCrosstalkSources_996715_31.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b54709b35241309db96e136e2ca6f4bc with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7e4c5b52-796e-491b-b3cd-23764137c4b0_isrForCrosstalkSources_996712_31.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/7e4c5b52-796e-491b-b3cd-23764137c4b0_isrForCrosstalkSources_996712_31.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dfbebe9bb7cc51a4001d64bc6f5a593 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fe08c529-e189-4bbb-8e23-56b85a244f98_isrForCrosstalkSources_996717_13.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/fe08c529-e189-4bbb-8e23-56b85a244f98_isrForCrosstalkSources_996717_13.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 402a38e5da4734d1212b5a08771d701c with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8e7615f8-b2a1-414b-aa92-ca6a1c1b3334_isrForCrosstalkSources_996717_25.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/8e7615f8-b2a1-414b-aa92-ca6a1c1b3334_isrForCrosstalkSources_996717_25.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e4c92cf9e29711df13817b919d0fafa with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f9e01117-56c2-4f39-aa31-6b345110aba3_isrForCrosstalkSources_996711_5.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f9e01117-56c2-4f39-aa31-6b345110aba3_isrForCrosstalkSources_996711_5.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c862d8d3ebb07822a7d370d3d39da20e with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/33fb22fc-5b5b-46c6-9b8c-2025109bd786_isrForCrosstalkSources_996710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/33fb22fc-5b5b-46c6-9b8c-2025109bd786_isrForCrosstalkSources_996710_55.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8802693bca05d7a90c056f8d5036eea with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ea193423-59d8-4e8f-8654-5756a2ec50ff_isrForCrosstalkSources_996714_35.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/ea193423-59d8-4e8f-8654-5756a2ec50ff_isrForCrosstalkSources_996714_35.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06b7babf40f87781fc3030ea1d045bb7 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/44a5b13c-48b7-49ba-96d8-6cab3934480c_isrForCrosstalkSources_996706_62.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/44a5b13c-48b7-49ba-96d8-6cab3934480c_isrForCrosstalkSources_996706_62.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36196f3a2a65ebd67347488df30e3c1c with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/2a584321-eff6-41c2-8df7-89974dba5fd3_isrForCrosstalkSources_996706_36.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/2a584321-eff6-41c2-8df7-89974dba5fd3_isrForCrosstalkSources_996706_36.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5552182579d5c1b08a64a5545baf7061 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/36cf8e2b-b8e5-4e3a-96e4-6f3aaa9c23c9_isrForCrosstalkSources_996721_59.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/36cf8e2b-b8e5-4e3a-96e4-6f3aaa9c23c9_isrForCrosstalkSources_996721_59.stderr 2024-03-15 12:19:50 proc_lsst.multi:170 [INFO] [multi] job_id 17071385 proc_lsst.multi INFO: [multi] job_id 17071385 2024-03-15 12:19:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-15 12:19:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071385 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071385 parsl.executors.status_handling DEBUG: Launched block 189 on executor multi with job ID 17071385 parsl.executors.status_handling INFO: Allocated block ID 190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-190 2024-03-15 12:19:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-15 12:19:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ec90da055fec061ece965c5f951ea9b with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/2148b870-693a-4d7b-8306-940ccb44c1cb_isrForCrosstalkSources_996716_47.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/2148b870-693a-4d7b-8306-940ccb44c1cb_isrForCrosstalkSources_996716_47.stderr 2024-03-15 12:19:51 proc_lsst.multi:170 [INFO] [multi] job_id 17071386 proc_lsst.multi INFO: [multi] job_id 17071386 2024-03-15 12:19:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-15 12:19:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071386 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071386 parsl.executors.status_handling DEBUG: Launched block 190 on executor multi with job ID 17071386 parsl.executors.status_handling INFO: Allocated block ID 191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-191 2024-03-15 12:19:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c8e359aad2c6c0be6300bc3402a6692 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/660ca067-3bde-4c6d-8f84-e20b92cad1f7_isrForCrosstalkSources_996707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/660ca067-3bde-4c6d-8f84-e20b92cad1f7_isrForCrosstalkSources_996707_3.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be48e837d23118dc0779f63548053590 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/fd283128-18b1-46de-b273-ead6e2983e4a_isrForCrosstalkSources_996716_13.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/fd283128-18b1-46de-b273-ead6e2983e4a_isrForCrosstalkSources_996716_13.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05c2d98ea66fefc3181e34d5a71bf8c9 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d89285be-3c54-4bae-9907-ef9c89ee5fe0_isrForCrosstalkSources_996709_54.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d89285be-3c54-4bae-9907-ef9c89ee5fe0_isrForCrosstalkSources_996709_54.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cd97ffaf8b6d6547920e0638a9a5897 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6249fc3e-4e15-495c-881b-5ee044e7b29d_isrForCrosstalkSources_996711_33.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/6249fc3e-4e15-495c-881b-5ee044e7b29d_isrForCrosstalkSources_996711_33.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bc5a5d6d3981fd9cbfa1e1ab08d099c with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/7ec36e98-1df4-4de2-a384-bbe1ef850662_isrForCrosstalkSources_996709_32.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/7ec36e98-1df4-4de2-a384-bbe1ef850662_isrForCrosstalkSources_996709_32.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4bafe3ba1ccadb55396b3422132346a with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/cb307ce3-6d54-4d46-8c8d-366b8d12c331_isrForCrosstalkSources_996709_47.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/cb307ce3-6d54-4d46-8c8d-366b8d12c331_isrForCrosstalkSources_996709_47.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ddc5f7a2175ac8ce95e1f0cf030b0da with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/16499b23-2ef0-4a01-adbe-56f4ba19a56b_isrForCrosstalkSources_996713_33.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/16499b23-2ef0-4a01-adbe-56f4ba19a56b_isrForCrosstalkSources_996713_33.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa42bd0e7ccdcfd077f37162f97b516c with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/e6e9ae94-dc6c-4105-af3f-75c743fb3515_isrForCrosstalkSources_996718_22.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/e6e9ae94-dc6c-4105-af3f-75c743fb3515_isrForCrosstalkSources_996718_22.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4960618dc1e0911dc6c291542b7ce9e9 with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5f8ed275-68a8-4d9e-89fe-11a8f5cda06e_isrForCrosstalkSources_996707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/5f8ed275-68a8-4d9e-89fe-11a8f5cda06e_isrForCrosstalkSources_996707_15.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d41400970e6309525291842ac114c35f with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/fc342a5c-4d6d-4c23-a8f8-2fb27e6c6c29_isrForCrosstalkSources_996715_19.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/fc342a5c-4d6d-4c23-a8f8-2fb27e6c6c29_isrForCrosstalkSources_996715_19.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 162ba29ff5e39852336389329c811e94 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/f739bd2d-520a-4566-9c61-ac94914de218_isrForCrosstalkSources_996721_25.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996721/f739bd2d-520a-4566-9c61-ac94914de218_isrForCrosstalkSources_996721_25.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca2c70c27ab6c1f9255628bfeb3d064d with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/24e83d4d-9b4e-4e48-bcb7-30ea7bd3eb38_isrForCrosstalkSources_996711_13.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/24e83d4d-9b4e-4e48-bcb7-30ea7bd3eb38_isrForCrosstalkSources_996711_13.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f59e618a33a36a72e94eda13256bffe with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/d37f6fef-22c7-499f-9938-b9d2f834e060_isrForCrosstalkSources_996716_24.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/d37f6fef-22c7-499f-9938-b9d2f834e060_isrForCrosstalkSources_996716_24.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fa4658520704b5f0f71ee8e4d66de09 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/9b3a1979-a70a-4754-883d-76d6dbdc42c5_isrForCrosstalkSources_996716_1.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996716/9b3a1979-a70a-4754-883d-76d6dbdc42c5_isrForCrosstalkSources_996716_1.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0d0f45c86f5cb46751ee4d2838bd82a with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e98f5307-509d-4b2c-8773-e617d79f33f1_isrForCrosstalkSources_996720_23.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/e98f5307-509d-4b2c-8773-e617d79f33f1_isrForCrosstalkSources_996720_23.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 926aa91e9e63ba5378d7ac32a5076b23 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/cc333598-ddc4-456e-aed5-3a25174a21df_isrForCrosstalkSources_996705_34.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996705/cc333598-ddc4-456e-aed5-3a25174a21df_isrForCrosstalkSources_996705_34.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d3d56bb5f7c0c7cfa14969e58e7dc3b with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6871ebeb-39f5-4ee9-842d-7335410eae21_isrForCrosstalkSources_996713_22.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/6871ebeb-39f5-4ee9-842d-7335410eae21_isrForCrosstalkSources_996713_22.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72a8ae6e65fb3ccbd2706cf6b5b63fbf with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d5c8b7e0-73ba-4e22-b326-07bd95205439_isrForCrosstalkSources_996709_12.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996709/d5c8b7e0-73ba-4e22-b326-07bd95205439_isrForCrosstalkSources_996709_12.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4991949864f2996255ffe4950b5f7fe7 with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f656146b-2d57-4fcc-b8c2-6417cef5b6e9_isrForCrosstalkSources_996711_25.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/f656146b-2d57-4fcc-b8c2-6417cef5b6e9_isrForCrosstalkSources_996711_25.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8e20b79a19136d5055e881272c560cf with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/94aafeb3-ec15-48fd-b144-8f105be80dd7_isrForCrosstalkSources_996718_57.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/94aafeb3-ec15-48fd-b144-8f105be80dd7_isrForCrosstalkSources_996718_57.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b725de2bfc572aeb80a1dcfa0ef564e with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/27a097e2-c283-4d40-98f7-c13d5e41ebd7_isrForCrosstalkSources_996706_53.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/27a097e2-c283-4d40-98f7-c13d5e41ebd7_isrForCrosstalkSources_996706_53.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48dcc35a2bb2fd70f1d40b02aa5df085 with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3a6cc474-ea2f-412c-ba45-e900eeae8472_isrForCrosstalkSources_996707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996707/3a6cc474-ea2f-412c-ba45-e900eeae8472_isrForCrosstalkSources_996707_29.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f3963c5ecc7c733e106e1393deb807a with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2523ec02-aa85-499c-982d-d11ad50f48fb_isrForCrosstalkSources_996713_50.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996713/2523ec02-aa85-499c-982d-d11ad50f48fb_isrForCrosstalkSources_996713_50.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26cc722e48d908d3dfaccfcf60830067 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5c29ba51-1dcd-4203-adda-a2a11f5e75ba_isrForCrosstalkSources_996719_1.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996719/5c29ba51-1dcd-4203-adda-a2a11f5e75ba_isrForCrosstalkSources_996719_1.stderr parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 790ba7f9f4fae431bd44c28c3517ffe2 with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f14c3509-de8a-417a-b031-5b99d1fa09ce_isrForCrosstalkSources_996717_45.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/f14c3509-de8a-417a-b031-5b99d1fa09ce_isrForCrosstalkSources_996717_45.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f913a50844b25414c16abd77ed5f060 with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/f24817c5-1af4-4a75-82a4-d0461e739a01_isrForCrosstalkSources_996720_40.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996720/f24817c5-1af4-4a75-82a4-d0461e739a01_isrForCrosstalkSources_996720_40.stderr parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd5857bb28bb804c8cf19e78c10d11fd with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/42857f6c-66f2-43c6-aecd-f9ac08dde79a_isrForCrosstalkSources_996706_48.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/42857f6c-66f2-43c6-aecd-f9ac08dde79a_isrForCrosstalkSources_996706_48.stderr 2024-03-15 12:19:51 proc_lsst.multi:170 [INFO] [multi] job_id 17071387 proc_lsst.multi INFO: [multi] job_id 17071387 2024-03-15 12:19:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-15 12:19:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071387 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071387 parsl.executors.status_handling DEBUG: Launched block 191 on executor multi with job ID 17071387 parsl.executors.status_handling INFO: Allocated block ID 192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-192 2024-03-15 12:19:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-15 12:19:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e21ef05c1cc789456a0a48ed3bb9cd92 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8bd57c42-063a-4a61-89ec-9b6857af725f_isrForCrosstalkSources_996706_2.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/8bd57c42-063a-4a61-89ec-9b6857af725f_isrForCrosstalkSources_996706_2.stderr 2024-03-15 12:19:52 proc_lsst.multi:170 [INFO] [multi] job_id 17071388 proc_lsst.multi INFO: [multi] job_id 17071388 2024-03-15 12:19:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-15 12:19:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071388 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071388 parsl.executors.status_handling DEBUG: Launched block 192 on executor multi with job ID 17071388 parsl.executors.status_handling INFO: Allocated block ID 193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-193 2024-03-15 12:19:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 839faadabf15e446c65b099a9b156c1b with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/470ede73-ff08-49d9-82d4-24ebb917cea1_isrForCrosstalkSources_996718_31.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/470ede73-ff08-49d9-82d4-24ebb917cea1_isrForCrosstalkSources_996718_31.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca0affed09335662b19cd3e66b7a1e8d with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/30571f27-f020-4fdc-89f2-daacc384ae9a_isrForCrosstalkSources_996710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/30571f27-f020-4fdc-89f2-daacc384ae9a_isrForCrosstalkSources_996710_47.stderr 2024-03-15 12:19:52 proc_lsst.multi:170 [INFO] [multi] job_id 17071389 proc_lsst.multi INFO: [multi] job_id 17071389 2024-03-15 12:19:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-15 12:19:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071389 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071389 parsl.executors.status_handling DEBUG: Launched block 193 on executor multi with job ID 17071389 parsl.executors.status_handling INFO: Allocated block ID 194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-194 2024-03-15 12:19:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-15 12:19:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e29d01910b692e526704290c4c24991a with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks 2024-03-15 12:19:52 proc_lsst.multi:170 [INFO] [multi] job_id 17071390 proc_lsst.multi INFO: [multi] job_id 17071390 2024-03-15 12:19:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-15 12:19:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071390 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071390 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1bc3a504-e82b-471c-831b-a5d50f9b9499_isrForCrosstalkSources_996717_30.stdout parsl.executors.status_handling DEBUG: Launched block 194 on executor multi with job ID 17071390 parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996717/1bc3a504-e82b-471c-831b-a5d50f9b9499_isrForCrosstalkSources_996717_30.stderr parsl.executors.status_handling INFO: Allocated block ID 195 parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-195 2024-03-15 12:19:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 parsl.dataflow.memoization DEBUG: Storing app cache entry 7eddb4ae70f1a4930e48b1ba298fccfc with result from task 708 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 parsl.dataflow.dflow DEBUG: Task 708 checkpointed proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/6a84d1cb-8796-4ec9-af51-75f342c9e716_isrForCrosstalkSources_996715_47.stdout proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996715/6a84d1cb-8796-4ec9-af51-75f342c9e716_isrForCrosstalkSources_996715_47.stderr proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:53 proc_lsst.multi:170 [INFO] [multi] job_id 17071391 proc_lsst.multi INFO: [multi] job_id 17071391 2024-03-15 12:19:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-15 12:19:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071391 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071391 parsl.executors.status_handling DEBUG: Launched block 195 on executor multi with job ID 17071391 parsl.executors.status_handling INFO: Allocated block ID 196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-196 2024-03-15 12:19:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:53 proc_lsst.multi:170 [INFO] [multi] job_id 17071392 proc_lsst.multi INFO: [multi] job_id 17071392 2024-03-15 12:19:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-15 12:19:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071392 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071392 parsl.executors.status_handling DEBUG: Launched block 196 on executor multi with job ID 17071392 parsl.executors.status_handling INFO: Allocated block ID 197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-197 2024-03-15 12:19:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-15 12:19:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:54 proc_lsst.multi:170 [INFO] [multi] job_id 17071393 proc_lsst.multi INFO: [multi] job_id 17071393 2024-03-15 12:19:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-15 12:19:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071393 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071393 parsl.executors.status_handling DEBUG: Launched block 197 on executor multi with job ID 17071393 parsl.executors.status_handling INFO: Allocated block ID 198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-198 2024-03-15 12:19:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:54 proc_lsst.multi:170 [INFO] [multi] job_id 17071395 proc_lsst.multi INFO: [multi] job_id 17071395 2024-03-15 12:19:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-15 12:19:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071395 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071395 parsl.executors.status_handling DEBUG: Launched block 198 on executor multi with job ID 17071395 parsl.executors.status_handling INFO: Allocated block ID 199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-199 2024-03-15 12:19:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-15 12:19:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:55 proc_lsst.multi:170 [INFO] [multi] job_id 17071396 proc_lsst.multi INFO: [multi] job_id 17071396 2024-03-15 12:19:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-15 12:19:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071396 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071396 parsl.executors.status_handling DEBUG: Launched block 199 on executor multi with job ID 17071396 parsl.executors.status_handling INFO: Allocated block ID 200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-200 2024-03-15 12:19:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:55 proc_lsst.multi:170 [INFO] [multi] job_id 17071397 proc_lsst.multi INFO: [multi] job_id 17071397 2024-03-15 12:19:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-15 12:19:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071397 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071397 parsl.executors.status_handling DEBUG: Launched block 200 on executor multi with job ID 17071397 parsl.executors.status_handling INFO: Allocated block ID 201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-201 2024-03-15 12:19:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-15 12:19:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:56 proc_lsst.multi:170 [INFO] [multi] job_id 17071398 proc_lsst.multi INFO: [multi] job_id 17071398 2024-03-15 12:19:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-15 12:19:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071398 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071398 parsl.executors.status_handling DEBUG: Launched block 201 on executor multi with job ID 17071398 parsl.executors.status_handling INFO: Allocated block ID 202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-202 2024-03-15 12:19:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:56 proc_lsst.multi:170 [INFO] [multi] job_id 17071399 proc_lsst.multi INFO: [multi] job_id 17071399 2024-03-15 12:19:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-15 12:19:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071399 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071399 parsl.executors.status_handling DEBUG: Launched block 202 on executor multi with job ID 17071399 parsl.executors.status_handling INFO: Allocated block ID 203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-203 2024-03-15 12:19:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:56 proc_lsst.multi:170 [INFO] [multi] job_id 17071401 proc_lsst.multi INFO: [multi] job_id 17071401 2024-03-15 12:19:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-15 12:19:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071401 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071401 parsl.executors.status_handling DEBUG: Launched block 203 on executor multi with job ID 17071401 parsl.executors.status_handling INFO: Allocated block ID 204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-204 2024-03-15 12:19:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-15 12:19:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:57 proc_lsst.multi:170 [INFO] [multi] job_id 17071402 proc_lsst.multi INFO: [multi] job_id 17071402 2024-03-15 12:19:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-15 12:19:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071402 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071402 parsl.executors.status_handling DEBUG: Launched block 204 on executor multi with job ID 17071402 parsl.executors.status_handling INFO: Allocated block ID 205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-205 2024-03-15 12:19:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:57 proc_lsst.multi:170 [INFO] [multi] job_id 17071403 proc_lsst.multi INFO: [multi] job_id 17071403 2024-03-15 12:19:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-15 12:19:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071403 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071403 parsl.executors.status_handling DEBUG: Launched block 205 on executor multi with job ID 17071403 parsl.executors.status_handling INFO: Allocated block ID 206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-206 2024-03-15 12:19:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-15 12:19:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:58 proc_lsst.multi:170 [INFO] [multi] job_id 17071404 proc_lsst.multi INFO: [multi] job_id 17071404 2024-03-15 12:19:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-15 12:19:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071404 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071404 parsl.executors.status_handling DEBUG: Launched block 206 on executor multi with job ID 17071404 parsl.executors.status_handling INFO: Allocated block ID 207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-207 2024-03-15 12:19:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:58 proc_lsst.multi:170 [INFO] [multi] job_id 17071405 proc_lsst.multi INFO: [multi] job_id 17071405 2024-03-15 12:19:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-15 12:19:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071405 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071405 parsl.executors.status_handling DEBUG: Launched block 207 on executor multi with job ID 17071405 parsl.executors.status_handling INFO: Allocated block ID 208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-208 2024-03-15 12:19:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-15 12:19:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:59 proc_lsst.multi:170 [INFO] [multi] job_id 17071406 proc_lsst.multi INFO: [multi] job_id 17071406 2024-03-15 12:19:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-15 12:19:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071406 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071406 parsl.executors.status_handling DEBUG: Launched block 208 on executor multi with job ID 17071406 parsl.executors.status_handling INFO: Allocated block ID 209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-209 2024-03-15 12:19:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:59 proc_lsst.multi:170 [INFO] [multi] job_id 17071407 proc_lsst.multi INFO: [multi] job_id 17071407 2024-03-15 12:19:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-15 12:19:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071407 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071407 parsl.executors.status_handling DEBUG: Launched block 209 on executor multi with job ID 17071407 parsl.executors.status_handling INFO: Allocated block ID 210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-210 2024-03-15 12:19:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:19:59 proc_lsst.multi:170 [INFO] [multi] job_id 17071408 proc_lsst.multi INFO: [multi] job_id 17071408 2024-03-15 12:19:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-15 12:19:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071408 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071408 parsl.executors.status_handling DEBUG: Launched block 210 on executor multi with job ID 17071408 parsl.executors.status_handling INFO: Allocated block ID 211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-211 2024-03-15 12:19:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:19:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:19:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-15 12:19:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:19:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:00 proc_lsst.multi:170 [INFO] [multi] job_id 17071409 proc_lsst.multi INFO: [multi] job_id 17071409 2024-03-15 12:20:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-15 12:20:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071409 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071409 parsl.executors.status_handling DEBUG: Launched block 211 on executor multi with job ID 17071409 parsl.executors.status_handling INFO: Allocated block ID 212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-212 2024-03-15 12:20:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 2024-03-15 12:20:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-15 12:20:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:02 proc_lsst.multi:170 [INFO] [multi] job_id 17071411 proc_lsst.multi INFO: [multi] job_id 17071411 2024-03-15 12:20:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-15 12:20:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071411 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071411 parsl.executors.status_handling DEBUG: Launched block 212 on executor multi with job ID 17071411 parsl.executors.status_handling INFO: Allocated block ID 213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-213 2024-03-15 12:20:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 2024-03-15 12:20:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-15 12:20:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:05 proc_lsst.multi:170 [INFO] [multi] job_id 17071413 proc_lsst.multi INFO: [multi] job_id 17071413 2024-03-15 12:20:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-15 12:20:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071413 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071413 parsl.executors.status_handling DEBUG: Launched block 213 on executor multi with job ID 17071413 parsl.executors.status_handling INFO: Allocated block ID 214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-214 2024-03-15 12:20:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:05 proc_lsst.multi:170 [INFO] [multi] job_id 17071415 proc_lsst.multi INFO: [multi] job_id 17071415 2024-03-15 12:20:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-15 12:20:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071415 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071415 parsl.executors.status_handling DEBUG: Launched block 214 on executor multi with job ID 17071415 parsl.executors.status_handling INFO: Allocated block ID 215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-215 2024-03-15 12:20:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:05 proc_lsst.multi:170 [INFO] [multi] job_id 17071416 proc_lsst.multi INFO: [multi] job_id 17071416 2024-03-15 12:20:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-15 12:20:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071416 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071416 parsl.executors.status_handling DEBUG: Launched block 215 on executor multi with job ID 17071416 parsl.executors.status_handling INFO: Allocated block ID 216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-216 2024-03-15 12:20:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-15 12:20:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b6815e96ea15ee0e17ca6c8e06c44e2 with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/8b0dff56-199f-4191-b8ae-3757fff179b3_isrForCrosstalkSources_996714_8.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996714/8b0dff56-199f-4191-b8ae-3757fff179b3_isrForCrosstalkSources_996714_8.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbeab97a133e0e122fec5cfb7386d109 with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/4365776c-45fd-4f4b-af82-174527b443ed_isrForCrosstalkSources_996712_13.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996712/4365776c-45fd-4f4b-af82-174527b443ed_isrForCrosstalkSources_996712_13.stderr parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2a48728672e21dd2c51d8999ecd155c with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9ac3aeb6-d2da-463d-8050-1b9186fa844f_isrForCrosstalkSources_996706_17.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996706/9ac3aeb6-d2da-463d-8050-1b9186fa844f_isrForCrosstalkSources_996706_17.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c44c70c3641c10683ae2b9c45353ed2 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/673a0b39-8e7f-4551-857f-bf39b7fc73ba_isrForCrosstalkSources_996710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/673a0b39-8e7f-4551-857f-bf39b7fc73ba_isrForCrosstalkSources_996710_38.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b71a78ff6ebb547636f9f2b672a74dc5 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9072f4db-f1f4-4dac-afb2-410c4362b8d8_isrForCrosstalkSources_996718_62.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/9072f4db-f1f4-4dac-afb2-410c4362b8d8_isrForCrosstalkSources_996718_62.stderr parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78f686d77999cd14a9358ef34230ebb1 with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cbafee00-b884-4c85-a1a5-035c879951ff_isrForCrosstalkSources_996710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996710/cbafee00-b884-4c85-a1a5-035c879951ff_isrForCrosstalkSources_996710_12.stderr parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b942394cb85eb17984fe0555ff272c96 with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/c7fc680d-0249-4e46-841a-eb5dde2613f8_isrForCrosstalkSources_996718_25.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996718/c7fc680d-0249-4e46-841a-eb5dde2613f8_isrForCrosstalkSources_996718_25.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89a04d155ca1f72c6aa931ae75f132d4 with result from task 838 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1054 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/86b56294-3f1a-4f3b-876a-dbef9771e45f_isrForCrosstalkSources_996711_34.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/logs/isrForCrosstalkSources/996711/86b56294-3f1a-4f3b-876a-dbef9771e45f_isrForCrosstalkSources_996711_34.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller 2024-03-15 12:20:06 proc_lsst.multi:170 [INFO] [multi] job_id 17071418 proc_lsst.multi INFO: [multi] job_id 17071418 2024-03-15 12:20:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-15 12:20:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071418 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071418 parsl.executors.status_handling DEBUG: Launched block 216 on executor multi with job ID 17071418 parsl.executors.status_handling INFO: Allocated block ID 217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-217 2024-03-15 12:20:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:06 proc_lsst.multi:170 [INFO] [multi] job_id 17071419 proc_lsst.multi INFO: [multi] job_id 17071419 2024-03-15 12:20:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-15 12:20:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071419 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071419 parsl.executors.status_handling DEBUG: Launched block 217 on executor multi with job ID 17071419 parsl.executors.status_handling INFO: Allocated block ID 218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-218 2024-03-15 12:20:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-15 12:20:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:07 proc_lsst.multi:170 [INFO] [multi] job_id 17071420 proc_lsst.multi INFO: [multi] job_id 17071420 2024-03-15 12:20:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-15 12:20:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071420 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071420 parsl.executors.status_handling DEBUG: Launched block 218 on executor multi with job ID 17071420 parsl.executors.status_handling INFO: Allocated block ID 219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-219 2024-03-15 12:20:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:07 proc_lsst.multi:170 [INFO] [multi] job_id 17071421 proc_lsst.multi INFO: [multi] job_id 17071421 2024-03-15 12:20:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-15 12:20:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071421 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071421 parsl.executors.status_handling DEBUG: Launched block 219 on executor multi with job ID 17071421 parsl.executors.status_handling INFO: Allocated block ID 220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-220 2024-03-15 12:20:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-15 12:20:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:08 proc_lsst.multi:170 [INFO] [multi] job_id 17071422 proc_lsst.multi INFO: [multi] job_id 17071422 2024-03-15 12:20:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-15 12:20:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071422 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071422 parsl.executors.status_handling DEBUG: Launched block 220 on executor multi with job ID 17071422 parsl.executors.status_handling INFO: Allocated block ID 221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-221 2024-03-15 12:20:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:08 proc_lsst.multi:170 [INFO] [multi] job_id 17071423 proc_lsst.multi INFO: [multi] job_id 17071423 2024-03-15 12:20:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-15 12:20:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071423 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071423 parsl.executors.status_handling DEBUG: Launched block 221 on executor multi with job ID 17071423 parsl.executors.status_handling INFO: Allocated block ID 222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-222 2024-03-15 12:20:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:08 proc_lsst.multi:170 [INFO] [multi] job_id 17071425 proc_lsst.multi INFO: [multi] job_id 17071425 2024-03-15 12:20:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-15 12:20:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071425 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071425 parsl.executors.status_handling DEBUG: Launched block 222 on executor multi with job ID 17071425 parsl.executors.status_handling INFO: Allocated block ID 223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-223 2024-03-15 12:20:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-15 12:20:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:09 proc_lsst.multi:170 [INFO] [multi] job_id 17071426 proc_lsst.multi INFO: [multi] job_id 17071426 2024-03-15 12:20:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-15 12:20:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071426 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071426 parsl.executors.status_handling DEBUG: Launched block 223 on executor multi with job ID 17071426 parsl.executors.status_handling INFO: Allocated block ID 224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-224 2024-03-15 12:20:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:09 proc_lsst.multi:170 [INFO] [multi] job_id 17071427 proc_lsst.multi INFO: [multi] job_id 17071427 2024-03-15 12:20:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-15 12:20:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071427 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071427 parsl.executors.status_handling DEBUG: Launched block 224 on executor multi with job ID 17071427 parsl.executors.status_handling INFO: Allocated block ID 225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-225 2024-03-15 12:20:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-15 12:20:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:10 proc_lsst.multi:170 [INFO] [multi] job_id 17071428 proc_lsst.multi INFO: [multi] job_id 17071428 2024-03-15 12:20:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-15 12:20:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071428 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071428 parsl.executors.status_handling DEBUG: Launched block 225 on executor multi with job ID 17071428 parsl.executors.status_handling INFO: Allocated block ID 226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-226 2024-03-15 12:20:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:10 proc_lsst.multi:170 [INFO] [multi] job_id 17071429 proc_lsst.multi INFO: [multi] job_id 17071429 2024-03-15 12:20:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-15 12:20:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071429 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071429 parsl.executors.status_handling DEBUG: Launched block 226 on executor multi with job ID 17071429 parsl.executors.status_handling INFO: Allocated block ID 227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-227 2024-03-15 12:20:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:10 proc_lsst.multi:170 [INFO] [multi] job_id 17071430 proc_lsst.multi INFO: [multi] job_id 17071430 2024-03-15 12:20:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-15 12:20:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071430 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071430 parsl.executors.status_handling DEBUG: Launched block 227 on executor multi with job ID 17071430 parsl.executors.status_handling INFO: Allocated block ID 228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-228 2024-03-15 12:20:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-15 12:20:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:11 proc_lsst.multi:170 [INFO] [multi] job_id 17071431 proc_lsst.multi INFO: [multi] job_id 17071431 2024-03-15 12:20:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-15 12:20:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071431 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071431 parsl.executors.status_handling DEBUG: Launched block 228 on executor multi with job ID 17071431 parsl.executors.status_handling INFO: Allocated block ID 229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-229 2024-03-15 12:20:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:11 proc_lsst.multi:170 [INFO] [multi] job_id 17071433 proc_lsst.multi INFO: [multi] job_id 17071433 2024-03-15 12:20:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-15 12:20:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071433 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071433 parsl.executors.status_handling DEBUG: Launched block 229 on executor multi with job ID 17071433 parsl.executors.status_handling INFO: Allocated block ID 230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-230 2024-03-15 12:20:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-15 12:20:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:12 proc_lsst.multi:170 [INFO] [multi] job_id 17071434 proc_lsst.multi INFO: [multi] job_id 17071434 2024-03-15 12:20:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-15 12:20:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071434 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071434 parsl.executors.status_handling DEBUG: Launched block 230 on executor multi with job ID 17071434 parsl.executors.status_handling INFO: Allocated block ID 231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-231 2024-03-15 12:20:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:12 proc_lsst.multi:170 [INFO] [multi] job_id 17071435 proc_lsst.multi INFO: [multi] job_id 17071435 2024-03-15 12:20:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-15 12:20:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071435 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071435 parsl.executors.status_handling DEBUG: Launched block 231 on executor multi with job ID 17071435 parsl.executors.status_handling INFO: Allocated block ID 232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-232 2024-03-15 12:20:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-15 12:20:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:13 proc_lsst.multi:170 [INFO] [multi] job_id 17071437 proc_lsst.multi INFO: [multi] job_id 17071437 2024-03-15 12:20:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 177 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 177 2024-03-15 12:20:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071437 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071437 parsl.executors.status_handling DEBUG: Launched block 232 on executor multi with job ID 17071437 parsl.executors.status_handling INFO: Allocated block ID 233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-233 2024-03-15 12:20:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 177 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 177 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:13 proc_lsst.multi:170 [INFO] [multi] job_id 17071445 proc_lsst.multi INFO: [multi] job_id 17071445 2024-03-15 12:20:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 178 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 178 2024-03-15 12:20:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071445 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071445 parsl.executors.status_handling DEBUG: Launched block 233 on executor multi with job ID 17071445 parsl.executors.status_handling INFO: Allocated block ID 234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-234 2024-03-15 12:20:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 178 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 178 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:13 proc_lsst.multi:170 [INFO] [multi] job_id 17071446 proc_lsst.multi INFO: [multi] job_id 17071446 2024-03-15 12:20:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 179 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 179 2024-03-15 12:20:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071446 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071446 parsl.executors.status_handling DEBUG: Launched block 234 on executor multi with job ID 17071446 parsl.executors.status_handling INFO: Allocated block ID 235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-235 2024-03-15 12:20:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 179 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 179 2024-03-15 12:20:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:14 proc_lsst.multi:170 [INFO] [multi] job_id 17071447 proc_lsst.multi INFO: [multi] job_id 17071447 2024-03-15 12:20:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 180 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 180 2024-03-15 12:20:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071447 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071447 parsl.executors.status_handling DEBUG: Launched block 235 on executor multi with job ID 17071447 parsl.executors.status_handling INFO: Allocated block ID 236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-236 2024-03-15 12:20:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 180 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 180 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:14 proc_lsst.multi:170 [INFO] [multi] job_id 17071448 proc_lsst.multi INFO: [multi] job_id 17071448 2024-03-15 12:20:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 181 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 181 2024-03-15 12:20:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071448 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071448 parsl.executors.status_handling DEBUG: Launched block 236 on executor multi with job ID 17071448 parsl.executors.status_handling INFO: Allocated block ID 237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-237 2024-03-15 12:20:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 181 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 181 2024-03-15 12:20:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:15 proc_lsst.multi:170 [INFO] [multi] job_id 17071449 proc_lsst.multi INFO: [multi] job_id 17071449 2024-03-15 12:20:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 182 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 182 2024-03-15 12:20:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071449 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071449 parsl.executors.status_handling DEBUG: Launched block 237 on executor multi with job ID 17071449 parsl.executors.status_handling INFO: Allocated block ID 238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-238 2024-03-15 12:20:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 182 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 182 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:15 proc_lsst.multi:170 [INFO] [multi] job_id 17071450 proc_lsst.multi INFO: [multi] job_id 17071450 2024-03-15 12:20:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 183 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 183 2024-03-15 12:20:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071450 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071450 parsl.executors.status_handling DEBUG: Launched block 238 on executor multi with job ID 17071450 parsl.executors.status_handling INFO: Allocated block ID 239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-239 2024-03-15 12:20:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 183 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 183 2024-03-15 12:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:16 proc_lsst.multi:170 [INFO] [multi] job_id 17071451 proc_lsst.multi INFO: [multi] job_id 17071451 2024-03-15 12:20:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 184 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 184 2024-03-15 12:20:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071451 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071451 parsl.executors.status_handling DEBUG: Launched block 239 on executor multi with job ID 17071451 parsl.executors.status_handling INFO: Allocated block ID 240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-240 2024-03-15 12:20:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 2024-03-15 12:20:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 12:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 12:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 12:20:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 12:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 12:20:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 12:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 12:20:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 12:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 12:20:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 12:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 184 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 184 2024-03-15 12:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,n3009,10.64.65.9,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54058 --result_port=54120 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210512/A0e/science#step0/20240315T121610Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 12:20:16 proc_lsst.multi:170 [INFO] [multi] job_id 17071452 proc_lsst.multi INFO: [multi] job_id 17071452 2024-03-15 12:20:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 185 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 185 2024-03-15 12:20:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 12:20:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17071452 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17071452 parsl.executors.status_handling DEBUG: Launched block 240 on executor multi with job ID 17071452 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23127545334096 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=241, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 98 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7fcc1279490 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7fcc1279490 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fce49db4737f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fce49db4737f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f8c66d22ff1a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f8c66d22ff1a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cc1d18d89300 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cc1d18d89300 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5ec570124081 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5ec570124081 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f4677e45a9df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f4677e45a9df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 94348306bbb7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 94348306bbb7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c6e88fc5c701 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c6e88fc5c701 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 40bbfa69d376 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 40bbfa69d376 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 40de376a318c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 40de376a318c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 779a9669476e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 779a9669476e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1b0319d2684a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1b0319d2684a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0af4c74284d6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0af4c74284d6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 155365876486 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 155365876486 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2323248093e3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2323248093e3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 763388eca228 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 763388eca228 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b347710726c8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b347710726c8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e1e242f04071 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e1e242f04071 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5669c29f72a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5669c29f72a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6e5b905ec6d2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6e5b905ec6d2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9e1e327f5b8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9e1e327f5b8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 031753ba529f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 031753ba529f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 416df01602d1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 416df01602d1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d548b9b74878 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d548b9b74878 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f83025d0511c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f83025d0511c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7701257d01f8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7701257d01f8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d65e7ec48d4a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d65e7ec48d4a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d591c923ffe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d591c923ffe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 64977acbc95f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 64977acbc95f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cba69e7621b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cba69e7621b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ea2b49a70b48 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ea2b49a70b48 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da2446fe6ad4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da2446fe6ad4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e147bc06320d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e147bc06320d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8c7df579315 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8c7df579315 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5c4b59fc575 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5c4b59fc575 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c48014182241 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c48014182241 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f1af2c169b28 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f1af2c169b28 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ac921ce24082 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ac921ce24082 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f03041736475 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f03041736475 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 40ca25e4dbf8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 40ca25e4dbf8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e2d3500b0ac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e2d3500b0ac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2087e3c553a1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2087e3c553a1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 29dedf78b36c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 29dedf78b36c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9bc121afddc6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9bc121afddc6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ed3f65883230 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ed3f65883230 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2da9ab88206d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2da9ab88206d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 005148dd24ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 005148dd24ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 48e0896c99b6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 48e0896c99b6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7196ad748fb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7196ad748fb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e86e315e62bb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e86e315e62bb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8f96f43201b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8f96f43201b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da93d8594ed0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da93d8594ed0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 76f6c63fe92b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 76f6c63fe92b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 232e59a7bffc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 232e59a7bffc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 826c964c5abe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 826c964c5abe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 797bd985568c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 797bd985568c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0bbd27a2dd11 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0bbd27a2dd11 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 32bc9861c6b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 32bc9861c6b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2dbd99e25c72 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2dbd99e25c72 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0874c07011c8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0874c07011c8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3f90ccaa7abe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3f90ccaa7abe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b03c27acbbb3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b03c27acbbb3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33e3343d2646 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33e3343d2646 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 39f664b9504d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 39f664b9504d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 35724955f646 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 35724955f646 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fa37ba2dfbb3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fa37ba2dfbb3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bd85c18e9d11 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bd85c18e9d11 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f4b0f433cd2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f4b0f433cd2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9bb0f8f8cd71 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9bb0f8f8cd71 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 71eb47665190 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 71eb47665190 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 59f6618842d5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 59f6618842d5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61b2f9d96db7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61b2f9d96db7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 938216fd6449 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 938216fd6449 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: babf4259a384 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: babf4259a384 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 693d2ee0ad1e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 693d2ee0ad1e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c67c3024e50 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c67c3024e50 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 842ee0f4cf47 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 842ee0f4cf47 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f6af2ffa17b6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f6af2ffa17b6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb877b2bb6f7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb877b2bb6f7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e72a11448be9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e72a11448be9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6f857156d813 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6f857156d813 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 855aa3d9ba96 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 855aa3d9ba96 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3c8c783b6df8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3c8c783b6df8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6f9dfbb01115 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6f9dfbb01115 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27be518d5b43 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27be518d5b43 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b44adef426e2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b44adef426e2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b248d08359ec parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b248d08359ec parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 095bc3f08be3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 095bc3f08be3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dcb88fce8ac0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dcb88fce8ac0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 363447f0c0e7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 363447f0c0e7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5386d940c17 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5386d940c17 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d69ff1169fc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d69ff1169fc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 48732a60a39a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 48732a60a39a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db5a01eeffd3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db5a01eeffd3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 07a95c4ff7df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 07a95c4ff7df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d6467b9032cd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d6467b9032cd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b69c3d2b11b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b69c3d2b11b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 04ae84ef6aba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 04ae84ef6aba 2024-03-15 12:20:16 proc_lsst.multi:146 [INFO] found job 17071274 in provider checkpoint proc_lsst.multi INFO: found job 17071274 in provider checkpoint 2024-03-15 12:20:16 proc_lsst.multi:201 [INFO] cancelling 17071274 on provider checkpoint proc_lsst.multi INFO: cancelling 17071274 on provider checkpoint 2024-03-15 12:20:17 proc_lsst.multi:146 [INFO] found job 17071272 in provider checkpoint proc_lsst.multi INFO: found job 17071272 in provider checkpoint 2024-03-15 12:20:17 proc_lsst.multi:201 [INFO] cancelling 17071272 on provider checkpoint proc_lsst.multi INFO: cancelling 17071272 on provider checkpoint 2024-03-15 12:20:17 proc_lsst.multi:146 [INFO] found job 17071270 in provider checkpoint proc_lsst.multi INFO: found job 17071270 in provider checkpoint 2024-03-15 12:20:17 proc_lsst.multi:201 [INFO] cancelling 17071270 on provider checkpoint proc_lsst.multi INFO: cancelling 17071270 on provider checkpoint 2024-03-15 12:20:17 proc_lsst.multi:146 [INFO] found job 17071271 in provider checkpoint proc_lsst.multi INFO: found job 17071271 in provider checkpoint 2024-03-15 12:20:17 proc_lsst.multi:201 [INFO] cancelling 17071271 on provider checkpoint proc_lsst.multi INFO: cancelling 17071271 on provider checkpoint 2024-03-15 12:20:18 proc_lsst.multi:146 [INFO] found job 17071264 in provider checkpoint proc_lsst.multi INFO: found job 17071264 in provider checkpoint 2024-03-15 12:20:18 proc_lsst.multi:201 [INFO] cancelling 17071264 on provider checkpoint proc_lsst.multi INFO: cancelling 17071264 on provider checkpoint 2024-03-15 12:20:18 proc_lsst.multi:146 [INFO] found job 17071259 in provider checkpoint proc_lsst.multi INFO: found job 17071259 in provider checkpoint 2024-03-15 12:20:18 proc_lsst.multi:201 [INFO] cancelling 17071259 on provider checkpoint proc_lsst.multi INFO: cancelling 17071259 on provider checkpoint 2024-03-15 12:20:19 proc_lsst.multi:146 [INFO] found job 17071261 in provider checkpoint proc_lsst.multi INFO: found job 17071261 in provider checkpoint 2024-03-15 12:20:19 proc_lsst.multi:201 [INFO] cancelling 17071261 on provider checkpoint proc_lsst.multi INFO: cancelling 17071261 on provider checkpoint 2024-03-15 12:20:19 proc_lsst.multi:146 [INFO] found job 17071266 in provider checkpoint proc_lsst.multi INFO: found job 17071266 in provider checkpoint 2024-03-15 12:20:19 proc_lsst.multi:201 [INFO] cancelling 17071266 on provider checkpoint proc_lsst.multi INFO: cancelling 17071266 on provider checkpoint 2024-03-15 12:20:20 proc_lsst.multi:146 [INFO] found job 17071263 in provider checkpoint proc_lsst.multi INFO: found job 17071263 in provider checkpoint 2024-03-15 12:20:20 proc_lsst.multi:201 [INFO] cancelling 17071263 on provider checkpoint proc_lsst.multi INFO: cancelling 17071263 on provider checkpoint 2024-03-15 12:20:20 proc_lsst.multi:146 [INFO] found job 17071257 in provider checkpoint proc_lsst.multi INFO: found job 17071257 in provider checkpoint 2024-03-15 12:20:20 proc_lsst.multi:201 [INFO] cancelling 17071257 on provider checkpoint proc_lsst.multi INFO: cancelling 17071257 on provider checkpoint 2024-03-15 12:20:20 proc_lsst.multi:146 [INFO] found job 17071275 in provider checkpoint proc_lsst.multi INFO: found job 17071275 in provider checkpoint 2024-03-15 12:20:20 proc_lsst.multi:201 [INFO] cancelling 17071275 on provider checkpoint proc_lsst.multi INFO: cancelling 17071275 on provider checkpoint 2024-03-15 12:20:21 proc_lsst.multi:146 [INFO] found job 17071250 in provider checkpoint proc_lsst.multi INFO: found job 17071250 in provider checkpoint 2024-03-15 12:20:21 proc_lsst.multi:201 [INFO] cancelling 17071250 on provider checkpoint proc_lsst.multi INFO: cancelling 17071250 on provider checkpoint 2024-03-15 12:20:21 proc_lsst.multi:146 [INFO] found job 17071258 in provider checkpoint proc_lsst.multi INFO: found job 17071258 in provider checkpoint 2024-03-15 12:20:21 proc_lsst.multi:201 [INFO] cancelling 17071258 on provider checkpoint proc_lsst.multi INFO: cancelling 17071258 on provider checkpoint 2024-03-15 12:20:22 proc_lsst.multi:146 [INFO] found job 17071276 in provider checkpoint proc_lsst.multi INFO: found job 17071276 in provider checkpoint 2024-03-15 12:20:22 proc_lsst.multi:201 [INFO] cancelling 17071276 on provider checkpoint proc_lsst.multi INFO: cancelling 17071276 on provider checkpoint 2024-03-15 12:20:22 proc_lsst.multi:146 [INFO] found job 17071273 in provider checkpoint proc_lsst.multi INFO: found job 17071273 in provider checkpoint 2024-03-15 12:20:22 proc_lsst.multi:201 [INFO] cancelling 17071273 on provider checkpoint proc_lsst.multi INFO: cancelling 17071273 on provider checkpoint 2024-03-15 12:20:23 proc_lsst.multi:146 [INFO] found job 17071265 in provider checkpoint proc_lsst.multi INFO: found job 17071265 in provider checkpoint 2024-03-15 12:20:23 proc_lsst.multi:201 [INFO] cancelling 17071265 on provider checkpoint proc_lsst.multi INFO: cancelling 17071265 on provider checkpoint 2024-03-15 12:20:23 proc_lsst.multi:146 [INFO] found job 17071277 in provider checkpoint proc_lsst.multi INFO: found job 17071277 in provider checkpoint 2024-03-15 12:20:23 proc_lsst.multi:201 [INFO] cancelling 17071277 on provider checkpoint proc_lsst.multi INFO: cancelling 17071277 on provider checkpoint 2024-03-15 12:20:23 proc_lsst.multi:146 [INFO] found job 17071279 in provider checkpoint proc_lsst.multi INFO: found job 17071279 in provider checkpoint 2024-03-15 12:20:23 proc_lsst.multi:201 [INFO] cancelling 17071279 on provider checkpoint proc_lsst.multi INFO: cancelling 17071279 on provider checkpoint 2024-03-15 12:20:24 proc_lsst.multi:146 [INFO] found job 17071278 in provider checkpoint proc_lsst.multi INFO: found job 17071278 in provider checkpoint 2024-03-15 12:20:24 proc_lsst.multi:201 [INFO] cancelling 17071278 on provider checkpoint proc_lsst.multi INFO: cancelling 17071278 on provider checkpoint 2024-03-15 12:20:24 proc_lsst.multi:146 [INFO] found job 17071280 in provider checkpoint proc_lsst.multi INFO: found job 17071280 in provider checkpoint 2024-03-15 12:20:24 proc_lsst.multi:201 [INFO] cancelling 17071280 on provider checkpoint proc_lsst.multi INFO: cancelling 17071280 on provider checkpoint 2024-03-15 12:20:25 proc_lsst.multi:146 [INFO] found job 91474 in provider local proc_lsst.multi INFO: found job 91474 in provider local 2024-03-15 12:20:25 proc_lsst.multi:201 [INFO] cancelling 91474 on provider local proc_lsst.multi INFO: cancelling 91474 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 91474 2024-03-15 12:20:25 proc_lsst.multi:146 [INFO] found job 17071157 in provider astro proc_lsst.multi INFO: found job 17071157 in provider astro 2024-03-15 12:20:25 proc_lsst.multi:201 [INFO] cancelling 17071157 on provider astro proc_lsst.multi INFO: cancelling 17071157 on provider astro 2024-03-15 12:20:25 proc_lsst.multi:146 [INFO] found job 17071284 in provider checkpoint proc_lsst.multi INFO: found job 17071284 in provider checkpoint 2024-03-15 12:20:25 proc_lsst.multi:201 [INFO] cancelling 17071284 on provider checkpoint proc_lsst.multi INFO: cancelling 17071284 on provider checkpoint 2024-03-15 12:20:25 proc_lsst.multi:146 [INFO] found job 17071136 in provider astro proc_lsst.multi INFO: found job 17071136 in provider astro 2024-03-15 12:20:25 proc_lsst.multi:201 [INFO] cancelling 17071136 on provider astro proc_lsst.multi INFO: cancelling 17071136 on provider astro 2024-03-15 12:20:26 proc_lsst.multi:146 [INFO] found job 17071125 in provider astro proc_lsst.multi INFO: found job 17071125 in provider astro 2024-03-15 12:20:26 proc_lsst.multi:201 [INFO] cancelling 17071125 on provider astro proc_lsst.multi INFO: cancelling 17071125 on provider astro 2024-03-15 12:20:26 proc_lsst.multi:146 [INFO] found job 17071148 in provider astro proc_lsst.multi INFO: found job 17071148 in provider astro 2024-03-15 12:20:26 proc_lsst.multi:201 [INFO] cancelling 17071148 on provider astro proc_lsst.multi INFO: cancelling 17071148 on provider astro 2024-03-15 12:20:27 proc_lsst.multi:146 [INFO] found job 17071133 in provider astro proc_lsst.multi INFO: found job 17071133 in provider astro 2024-03-15 12:20:27 proc_lsst.multi:201 [INFO] cancelling 17071133 on provider astro proc_lsst.multi INFO: cancelling 17071133 on provider astro 2024-03-15 12:20:27 proc_lsst.multi:146 [INFO] found job 17071137 in provider astro proc_lsst.multi INFO: found job 17071137 in provider astro 2024-03-15 12:20:27 proc_lsst.multi:201 [INFO] cancelling 17071137 on provider astro proc_lsst.multi INFO: cancelling 17071137 on provider astro 2024-03-15 12:20:27 proc_lsst.multi:146 [INFO] found job 17071135 in provider astro proc_lsst.multi INFO: found job 17071135 in provider astro 2024-03-15 12:20:27 proc_lsst.multi:201 [INFO] cancelling 17071135 on provider astro proc_lsst.multi INFO: cancelling 17071135 on provider astro 2024-03-15 12:20:28 proc_lsst.multi:146 [INFO] found job 17071149 in provider astro proc_lsst.multi INFO: found job 17071149 in provider astro 2024-03-15 12:20:28 proc_lsst.multi:201 [INFO] cancelling 17071149 on provider astro proc_lsst.multi INFO: cancelling 17071149 on provider astro 2024-03-15 12:20:28 proc_lsst.multi:146 [INFO] found job 17071151 in provider astro proc_lsst.multi INFO: found job 17071151 in provider astro 2024-03-15 12:20:28 proc_lsst.multi:201 [INFO] cancelling 17071151 on provider astro proc_lsst.multi INFO: cancelling 17071151 on provider astro 2024-03-15 12:20:29 proc_lsst.multi:146 [INFO] found job 17071160 in provider astro proc_lsst.multi INFO: found job 17071160 in provider astro 2024-03-15 12:20:29 proc_lsst.multi:201 [INFO] cancelling 17071160 on provider astro proc_lsst.multi INFO: cancelling 17071160 on provider astro 2024-03-15 12:20:29 proc_lsst.multi:146 [INFO] found job 17071141 in provider astro proc_lsst.multi INFO: found job 17071141 in provider astro 2024-03-15 12:20:29 proc_lsst.multi:201 [INFO] cancelling 17071141 on provider astro proc_lsst.multi INFO: cancelling 17071141 on provider astro 2024-03-15 12:20:29 proc_lsst.multi:146 [INFO] found job 17071152 in provider astro proc_lsst.multi INFO: found job 17071152 in provider astro 2024-03-15 12:20:29 proc_lsst.multi:201 [INFO] cancelling 17071152 on provider astro proc_lsst.multi INFO: cancelling 17071152 on provider astro 2024-03-15 12:20:30 proc_lsst.multi:146 [INFO] found job 17071143 in provider astro proc_lsst.multi INFO: found job 17071143 in provider astro 2024-03-15 12:20:30 proc_lsst.multi:201 [INFO] cancelling 17071143 on provider astro proc_lsst.multi INFO: cancelling 17071143 on provider astro 2024-03-15 12:20:30 proc_lsst.multi:146 [INFO] found job 17071139 in provider astro proc_lsst.multi INFO: found job 17071139 in provider astro 2024-03-15 12:20:30 proc_lsst.multi:201 [INFO] cancelling 17071139 on provider astro proc_lsst.multi INFO: cancelling 17071139 on provider astro 2024-03-15 12:20:31 proc_lsst.multi:146 [INFO] found job 17071132 in provider astro proc_lsst.multi INFO: found job 17071132 in provider astro 2024-03-15 12:20:31 proc_lsst.multi:201 [INFO] cancelling 17071132 on provider astro proc_lsst.multi INFO: cancelling 17071132 on provider astro 2024-03-15 12:20:31 proc_lsst.multi:146 [INFO] found job 17071142 in provider astro proc_lsst.multi INFO: found job 17071142 in provider astro 2024-03-15 12:20:31 proc_lsst.multi:201 [INFO] cancelling 17071142 on provider astro proc_lsst.multi INFO: cancelling 17071142 on provider astro 2024-03-15 12:20:31 proc_lsst.multi:146 [INFO] found job 17071243 in provider checkpoint proc_lsst.multi INFO: found job 17071243 in provider checkpoint 2024-03-15 12:20:31 proc_lsst.multi:201 [INFO] cancelling 17071243 on provider checkpoint proc_lsst.multi INFO: cancelling 17071243 on provider checkpoint 2024-03-15 12:20:32 proc_lsst.multi:146 [INFO] found job 17071124 in provider astro proc_lsst.multi INFO: found job 17071124 in provider astro 2024-03-15 12:20:32 proc_lsst.multi:201 [INFO] cancelling 17071124 on provider astro proc_lsst.multi INFO: cancelling 17071124 on provider astro 2024-03-15 12:20:32 proc_lsst.multi:146 [INFO] found job 17071269 in provider checkpoint proc_lsst.multi INFO: found job 17071269 in provider checkpoint 2024-03-15 12:20:32 proc_lsst.multi:201 [INFO] cancelling 17071269 on provider checkpoint proc_lsst.multi INFO: cancelling 17071269 on provider checkpoint 2024-03-15 12:20:33 proc_lsst.multi:146 [INFO] found job 17071245 in provider checkpoint proc_lsst.multi INFO: found job 17071245 in provider checkpoint 2024-03-15 12:20:33 proc_lsst.multi:201 [INFO] cancelling 17071245 on provider checkpoint proc_lsst.multi INFO: cancelling 17071245 on provider checkpoint 2024-03-15 12:20:33 proc_lsst.multi:146 [INFO] found job 17071249 in provider checkpoint proc_lsst.multi INFO: found job 17071249 in provider checkpoint 2024-03-15 12:20:33 proc_lsst.multi:201 [INFO] cancelling 17071249 on provider checkpoint proc_lsst.multi INFO: cancelling 17071249 on provider checkpoint 2024-03-15 12:20:33 proc_lsst.multi:146 [INFO] found job 17071244 in provider checkpoint proc_lsst.multi INFO: found job 17071244 in provider checkpoint 2024-03-15 12:20:33 proc_lsst.multi:201 [INFO] cancelling 17071244 on provider checkpoint proc_lsst.multi INFO: cancelling 17071244 on provider checkpoint 2024-03-15 12:20:34 proc_lsst.multi:146 [INFO] found job 17071232 in provider checkpoint proc_lsst.multi INFO: found job 17071232 in provider checkpoint 2024-03-15 12:20:34 proc_lsst.multi:201 [INFO] cancelling 17071232 on provider checkpoint proc_lsst.multi INFO: cancelling 17071232 on provider checkpoint 2024-03-15 12:20:34 proc_lsst.multi:146 [INFO] found job 17071240 in provider checkpoint proc_lsst.multi INFO: found job 17071240 in provider checkpoint 2024-03-15 12:20:34 proc_lsst.multi:201 [INFO] cancelling 17071240 on provider checkpoint proc_lsst.multi INFO: cancelling 17071240 on provider checkpoint 2024-03-15 12:20:35 proc_lsst.multi:146 [INFO] found job 17071251 in provider checkpoint proc_lsst.multi INFO: found job 17071251 in provider checkpoint 2024-03-15 12:20:35 proc_lsst.multi:201 [INFO] cancelling 17071251 on provider checkpoint proc_lsst.multi INFO: cancelling 17071251 on provider checkpoint 2024-03-15 12:20:35 proc_lsst.multi:146 [INFO] found job 17071215 in provider escience_a40 proc_lsst.multi INFO: found job 17071215 in provider escience_a40 2024-03-15 12:20:35 proc_lsst.multi:201 [INFO] cancelling 17071215 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071215 on provider escience_a40 2024-03-15 12:20:35 proc_lsst.multi:146 [INFO] found job 17071131 in provider astro proc_lsst.multi INFO: found job 17071131 in provider astro 2024-03-15 12:20:35 proc_lsst.multi:201 [INFO] cancelling 17071131 on provider astro proc_lsst.multi INFO: cancelling 17071131 on provider astro 2024-03-15 12:20:37 proc_lsst.multi:146 [INFO] found job 17071220 in provider escience_a40 proc_lsst.multi INFO: found job 17071220 in provider escience_a40 2024-03-15 12:20:37 proc_lsst.multi:201 [INFO] cancelling 17071220 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071220 on provider escience_a40 2024-03-15 12:20:38 proc_lsst.multi:146 [INFO] found job 17071162 in provider astro proc_lsst.multi INFO: found job 17071162 in provider astro 2024-03-15 12:20:38 proc_lsst.multi:201 [INFO] cancelling 17071162 on provider astro proc_lsst.multi INFO: cancelling 17071162 on provider astro 2024-03-15 12:20:38 proc_lsst.multi:146 [INFO] found job 17071130 in provider astro proc_lsst.multi INFO: found job 17071130 in provider astro 2024-03-15 12:20:38 proc_lsst.multi:201 [INFO] cancelling 17071130 on provider astro proc_lsst.multi INFO: cancelling 17071130 on provider astro 2024-03-15 12:20:39 proc_lsst.multi:146 [INFO] found job 17071126 in provider astro proc_lsst.multi INFO: found job 17071126 in provider astro 2024-03-15 12:20:39 proc_lsst.multi:201 [INFO] cancelling 17071126 on provider astro proc_lsst.multi INFO: cancelling 17071126 on provider astro 2024-03-15 12:20:39 proc_lsst.multi:146 [INFO] found job 17071127 in provider astro proc_lsst.multi INFO: found job 17071127 in provider astro 2024-03-15 12:20:39 proc_lsst.multi:201 [INFO] cancelling 17071127 on provider astro proc_lsst.multi INFO: cancelling 17071127 on provider astro 2024-03-15 12:20:40 proc_lsst.multi:146 [INFO] found job 17071128 in provider astro proc_lsst.multi INFO: found job 17071128 in provider astro 2024-03-15 12:20:40 proc_lsst.multi:201 [INFO] cancelling 17071128 on provider astro proc_lsst.multi INFO: cancelling 17071128 on provider astro 2024-03-15 12:20:40 proc_lsst.multi:146 [INFO] found job 17071156 in provider astro proc_lsst.multi INFO: found job 17071156 in provider astro 2024-03-15 12:20:40 proc_lsst.multi:201 [INFO] cancelling 17071156 on provider astro proc_lsst.multi INFO: cancelling 17071156 on provider astro 2024-03-15 12:20:40 proc_lsst.multi:146 [INFO] found job 17071213 in provider escience_a40 proc_lsst.multi INFO: found job 17071213 in provider escience_a40 2024-03-15 12:20:40 proc_lsst.multi:201 [INFO] cancelling 17071213 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071213 on provider escience_a40 2024-03-15 12:20:41 proc_lsst.multi:146 [INFO] found job 17071253 in provider checkpoint proc_lsst.multi INFO: found job 17071253 in provider checkpoint 2024-03-15 12:20:41 proc_lsst.multi:201 [INFO] cancelling 17071253 on provider checkpoint proc_lsst.multi INFO: cancelling 17071253 on provider checkpoint 2024-03-15 12:20:41 proc_lsst.multi:146 [INFO] found job 17071237 in provider checkpoint proc_lsst.multi INFO: found job 17071237 in provider checkpoint 2024-03-15 12:20:41 proc_lsst.multi:201 [INFO] cancelling 17071237 on provider checkpoint proc_lsst.multi INFO: cancelling 17071237 on provider checkpoint 2024-03-15 12:20:41 proc_lsst.multi:146 [INFO] found job 17071211 in provider escience_a40 proc_lsst.multi INFO: found job 17071211 in provider escience_a40 2024-03-15 12:20:41 proc_lsst.multi:201 [INFO] cancelling 17071211 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071211 on provider escience_a40 2024-03-15 12:20:42 proc_lsst.multi:146 [INFO] found job 17071252 in provider checkpoint proc_lsst.multi INFO: found job 17071252 in provider checkpoint 2024-03-15 12:20:42 proc_lsst.multi:201 [INFO] cancelling 17071252 on provider checkpoint proc_lsst.multi INFO: cancelling 17071252 on provider checkpoint 2024-03-15 12:20:42 proc_lsst.multi:146 [INFO] found job 17071166 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071166 in provider escience_rtx6k 2024-03-15 12:20:42 proc_lsst.multi:201 [INFO] cancelling 17071166 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071166 on provider escience_rtx6k 2024-03-15 12:20:43 proc_lsst.multi:146 [INFO] found job 17071144 in provider astro proc_lsst.multi INFO: found job 17071144 in provider astro 2024-03-15 12:20:43 proc_lsst.multi:201 [INFO] cancelling 17071144 on provider astro proc_lsst.multi INFO: cancelling 17071144 on provider astro 2024-03-15 12:20:43 proc_lsst.multi:146 [INFO] found job 17071225 in provider escience_a40 proc_lsst.multi INFO: found job 17071225 in provider escience_a40 2024-03-15 12:20:43 proc_lsst.multi:201 [INFO] cancelling 17071225 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071225 on provider escience_a40 2024-03-15 12:20:43 proc_lsst.multi:146 [INFO] found job 17071222 in provider escience_a40 proc_lsst.multi INFO: found job 17071222 in provider escience_a40 2024-03-15 12:20:43 proc_lsst.multi:201 [INFO] cancelling 17071222 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071222 on provider escience_a40 2024-03-15 12:20:44 proc_lsst.multi:146 [INFO] found job 17071212 in provider escience_a40 proc_lsst.multi INFO: found job 17071212 in provider escience_a40 2024-03-15 12:20:44 proc_lsst.multi:201 [INFO] cancelling 17071212 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071212 on provider escience_a40 2024-03-15 12:20:44 proc_lsst.multi:146 [INFO] found job 17071134 in provider astro proc_lsst.multi INFO: found job 17071134 in provider astro 2024-03-15 12:20:44 proc_lsst.multi:201 [INFO] cancelling 17071134 on provider astro proc_lsst.multi INFO: cancelling 17071134 on provider astro 2024-03-15 12:20:44 proc_lsst.multi:146 [INFO] found job 17071231 in provider checkpoint proc_lsst.multi INFO: found job 17071231 in provider checkpoint 2024-03-15 12:20:44 proc_lsst.multi:201 [INFO] cancelling 17071231 on provider checkpoint proc_lsst.multi INFO: cancelling 17071231 on provider checkpoint 2024-03-15 12:20:45 proc_lsst.multi:146 [INFO] found job 17071158 in provider astro proc_lsst.multi INFO: found job 17071158 in provider astro 2024-03-15 12:20:45 proc_lsst.multi:201 [INFO] cancelling 17071158 on provider astro proc_lsst.multi INFO: cancelling 17071158 on provider astro 2024-03-15 12:20:45 proc_lsst.multi:146 [INFO] found job 17071210 in provider escience_a40 proc_lsst.multi INFO: found job 17071210 in provider escience_a40 2024-03-15 12:20:45 proc_lsst.multi:201 [INFO] cancelling 17071210 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071210 on provider escience_a40 2024-03-15 12:20:46 proc_lsst.multi:146 [INFO] found job 17071154 in provider astro proc_lsst.multi INFO: found job 17071154 in provider astro 2024-03-15 12:20:46 proc_lsst.multi:201 [INFO] cancelling 17071154 on provider astro proc_lsst.multi INFO: cancelling 17071154 on provider astro 2024-03-15 12:20:46 proc_lsst.multi:146 [INFO] found job 17071153 in provider astro proc_lsst.multi INFO: found job 17071153 in provider astro 2024-03-15 12:20:46 proc_lsst.multi:201 [INFO] cancelling 17071153 on provider astro proc_lsst.multi INFO: cancelling 17071153 on provider astro 2024-03-15 12:20:46 proc_lsst.multi:146 [INFO] found job 17071164 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071164 in provider escience_rtx6k 2024-03-15 12:20:46 proc_lsst.multi:201 [INFO] cancelling 17071164 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071164 on provider escience_rtx6k 2024-03-15 12:20:47 proc_lsst.multi:146 [INFO] found job 17071229 in provider escience_a40 proc_lsst.multi INFO: found job 17071229 in provider escience_a40 2024-03-15 12:20:47 proc_lsst.multi:201 [INFO] cancelling 17071229 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071229 on provider escience_a40 2024-03-15 12:20:47 proc_lsst.multi:146 [INFO] found job 17071235 in provider checkpoint proc_lsst.multi INFO: found job 17071235 in provider checkpoint 2024-03-15 12:20:47 proc_lsst.multi:201 [INFO] cancelling 17071235 on provider checkpoint proc_lsst.multi INFO: cancelling 17071235 on provider checkpoint 2024-03-15 12:20:48 proc_lsst.multi:146 [INFO] found job 17071219 in provider escience_a40 proc_lsst.multi INFO: found job 17071219 in provider escience_a40 2024-03-15 12:20:48 proc_lsst.multi:201 [INFO] cancelling 17071219 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071219 on provider escience_a40 2024-03-15 12:20:48 proc_lsst.multi:146 [INFO] found job 17071209 in provider escience_a40 proc_lsst.multi INFO: found job 17071209 in provider escience_a40 2024-03-15 12:20:48 proc_lsst.multi:201 [INFO] cancelling 17071209 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071209 on provider escience_a40 2024-03-15 12:20:48 proc_lsst.multi:146 [INFO] found job 17071224 in provider escience_a40 proc_lsst.multi INFO: found job 17071224 in provider escience_a40 2024-03-15 12:20:48 proc_lsst.multi:201 [INFO] cancelling 17071224 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071224 on provider escience_a40 2024-03-15 12:20:49 proc_lsst.multi:146 [INFO] found job 17071165 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071165 in provider escience_rtx6k 2024-03-15 12:20:49 proc_lsst.multi:201 [INFO] cancelling 17071165 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071165 on provider escience_rtx6k 2024-03-15 12:20:49 proc_lsst.multi:146 [INFO] found job 17071218 in provider escience_a40 proc_lsst.multi INFO: found job 17071218 in provider escience_a40 2024-03-15 12:20:49 proc_lsst.multi:201 [INFO] cancelling 17071218 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071218 on provider escience_a40 2024-03-15 12:20:50 proc_lsst.multi:146 [INFO] found job 17071217 in provider escience_a40 proc_lsst.multi INFO: found job 17071217 in provider escience_a40 2024-03-15 12:20:50 proc_lsst.multi:201 [INFO] cancelling 17071217 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071217 on provider escience_a40 2024-03-15 12:20:50 proc_lsst.multi:146 [INFO] found job 17071226 in provider escience_a40 proc_lsst.multi INFO: found job 17071226 in provider escience_a40 2024-03-15 12:20:50 proc_lsst.multi:201 [INFO] cancelling 17071226 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071226 on provider escience_a40 2024-03-15 12:20:51 proc_lsst.multi:146 [INFO] found job 17071214 in provider escience_a40 proc_lsst.multi INFO: found job 17071214 in provider escience_a40 2024-03-15 12:20:51 proc_lsst.multi:201 [INFO] cancelling 17071214 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071214 on provider escience_a40 2024-03-15 12:20:51 proc_lsst.multi:146 [INFO] found job 17071216 in provider escience_a40 proc_lsst.multi INFO: found job 17071216 in provider escience_a40 2024-03-15 12:20:51 proc_lsst.multi:201 [INFO] cancelling 17071216 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071216 on provider escience_a40 2024-03-15 12:20:51 proc_lsst.multi:146 [INFO] found job 17071221 in provider escience_a40 proc_lsst.multi INFO: found job 17071221 in provider escience_a40 2024-03-15 12:20:51 proc_lsst.multi:201 [INFO] cancelling 17071221 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071221 on provider escience_a40 2024-03-15 12:20:52 proc_lsst.multi:146 [INFO] found job 17071163 in provider astro proc_lsst.multi INFO: found job 17071163 in provider astro 2024-03-15 12:20:52 proc_lsst.multi:201 [INFO] cancelling 17071163 on provider astro proc_lsst.multi INFO: cancelling 17071163 on provider astro 2024-03-15 12:20:52 proc_lsst.multi:146 [INFO] found job 17071247 in provider checkpoint proc_lsst.multi INFO: found job 17071247 in provider checkpoint 2024-03-15 12:20:52 proc_lsst.multi:201 [INFO] cancelling 17071247 on provider checkpoint proc_lsst.multi INFO: cancelling 17071247 on provider checkpoint 2024-03-15 12:20:53 proc_lsst.multi:146 [INFO] found job 17071248 in provider checkpoint proc_lsst.multi INFO: found job 17071248 in provider checkpoint 2024-03-15 12:20:53 proc_lsst.multi:201 [INFO] cancelling 17071248 on provider checkpoint proc_lsst.multi INFO: cancelling 17071248 on provider checkpoint 2024-03-15 12:20:53 proc_lsst.multi:146 [INFO] found job 17071230 in provider checkpoint proc_lsst.multi INFO: found job 17071230 in provider checkpoint 2024-03-15 12:20:53 proc_lsst.multi:201 [INFO] cancelling 17071230 on provider checkpoint proc_lsst.multi INFO: cancelling 17071230 on provider checkpoint 2024-03-15 12:20:53 proc_lsst.multi:146 [INFO] found job 17071228 in provider escience_a40 proc_lsst.multi INFO: found job 17071228 in provider escience_a40 2024-03-15 12:20:53 proc_lsst.multi:201 [INFO] cancelling 17071228 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071228 on provider escience_a40 2024-03-15 12:20:54 proc_lsst.multi:146 [INFO] found job 17071227 in provider escience_a40 proc_lsst.multi INFO: found job 17071227 in provider escience_a40 2024-03-15 12:20:54 proc_lsst.multi:201 [INFO] cancelling 17071227 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071227 on provider escience_a40 2024-03-15 12:20:54 proc_lsst.multi:146 [INFO] found job 17071239 in provider checkpoint proc_lsst.multi INFO: found job 17071239 in provider checkpoint 2024-03-15 12:20:54 proc_lsst.multi:201 [INFO] cancelling 17071239 on provider checkpoint proc_lsst.multi INFO: cancelling 17071239 on provider checkpoint 2024-03-15 12:20:54 proc_lsst.multi:146 [INFO] found job 17071267 in provider checkpoint proc_lsst.multi INFO: found job 17071267 in provider checkpoint 2024-03-15 12:20:54 proc_lsst.multi:201 [INFO] cancelling 17071267 on provider checkpoint proc_lsst.multi INFO: cancelling 17071267 on provider checkpoint 2024-03-15 12:20:55 proc_lsst.multi:146 [INFO] found job 17071238 in provider checkpoint proc_lsst.multi INFO: found job 17071238 in provider checkpoint 2024-03-15 12:20:55 proc_lsst.multi:201 [INFO] cancelling 17071238 on provider checkpoint proc_lsst.multi INFO: cancelling 17071238 on provider checkpoint 2024-03-15 12:20:55 proc_lsst.multi:146 [INFO] found job 17071199 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071199 in provider escience_rtx6k 2024-03-15 12:20:55 proc_lsst.multi:201 [INFO] cancelling 17071199 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071199 on provider escience_rtx6k 2024-03-15 12:20:56 proc_lsst.multi:146 [INFO] found job 17071268 in provider checkpoint proc_lsst.multi INFO: found job 17071268 in provider checkpoint 2024-03-15 12:20:56 proc_lsst.multi:201 [INFO] cancelling 17071268 on provider checkpoint proc_lsst.multi INFO: cancelling 17071268 on provider checkpoint 2024-03-15 12:20:56 proc_lsst.multi:146 [INFO] found job 17071208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071208 in provider escience_rtx6k 2024-03-15 12:20:56 proc_lsst.multi:201 [INFO] cancelling 17071208 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071208 on provider escience_rtx6k 2024-03-15 12:20:56 proc_lsst.multi:146 [INFO] found job 17071281 in provider checkpoint proc_lsst.multi INFO: found job 17071281 in provider checkpoint 2024-03-15 12:20:56 proc_lsst.multi:201 [INFO] cancelling 17071281 on provider checkpoint proc_lsst.multi INFO: cancelling 17071281 on provider checkpoint parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-15 12:20:57 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-15 12:20:57 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17071230', '17071231', '17071232', '17071235', '17071237', '17071238', '17071239', '17071240', '17071243', '17071244', '17071245', '17071247', '17071248', '17071249', '17071250', '17071251', '17071252', '17071253', '17071257', '17071258', '17071259', '17071261', '17071263', '17071264', '17071265', '17071266', '17071267', '17071268', '17071269', '17071270', '17071271', '17071272', '17071273', '17071274', '17071275', '17071276', '17071277', '17071278', '17071279', '17071280', '17071281', '17071284', '17071285', '17071286', '17071288', '17071289', '17071290', '17071291', '17071292', '17071293', '17071294', '17071295', '17071296', '17071297', '17071298', '17071299', '17071300', '17071301', '17071302', '17071303', '17071304', '17071305', '17071306', '17071307', '17071308', '17071309', '17071310', '17071311', '17071312', '17071313', '17071314', '17071315', '17071316', '17071317', '17071318', '17071319', '17071320', '17071321', '17071322', '17071323', '17071324', '17071325', '17071326', '17071328', '17071329', '17071330', '17071331', '17071334', '17071335', '17071336', '17071337', '17071338', '17071339', '17071340', '17071341', '17071342', '17071343', '17071344', '17071345', '17071346', '17071347', '17071348', '17071349', '17071350', '17071351', '17071352', '17071353', '17071355', '17071356', '17071357', '17071359', '17071360', '17071361', '17071362', '17071363', '17071365', '17071366', '17071367', '17071368', '17071369', '17071370', '17071372', '17071373', '17071374', '17071376', '17071377', '17071378', '17071379', '17071380', '17071381', '17071382', '17071383', '17071384', '17071385', '17071386', '17071387', '17071388', '17071389', '17071390', '17071391', '17071392', '17071393', '17071395', '17071396', '17071397', '17071398', '17071399', '17071401', '17071402', '17071403', '17071404', '17071405', '17071406', '17071407', '17071408', '17071409', '17071411', '17071413', '17071415', '17071416', '17071418', '17071419', '17071420', '17071421', '17071422', '17071423', '17071425', '17071426', '17071427', '17071428', '17071429', '17071430', '17071431', '17071433', '17071434', '17071435', '17071437', '17071445', '17071446', '17071447', '17071448', '17071449', '17071450', '17071451', '17071452', '17071124', '17071125', '17071126', '17071127', '17071128', '17071130', '17071131', '17071132', '17071133', '17071134', '17071135', '17071136', '17071137', '17071139', '17071141', '17071142', '17071143', '17071144', '17071148', '17071149', '17071151', '17071152', '17071153', '17071154', '17071156', '17071157', '17071158', '17071160', '17071162', '17071163', '17071209', '17071210', '17071211', '17071212', '17071213', '17071214', '17071215', '17071216', '17071217', '17071218', '17071219', '17071220', '17071221', '17071222', '17071224', '17071225', '17071226', '17071227', '17071228', '17071229', '17071164', '17071165', '17071166', '17071199', '17071208', '91474'] proc_lsst.multi INFO: new jobs since last cancel ['17071230', '17071231', '17071232', '17071235', '17071237', '17071238', '17071239', '17071240', '17071243', '17071244', '17071245', '17071247', '17071248', '17071249', '17071250', '17071251', '17071252', '17071253', '17071257', '17071258', '17071259', '17071261', '17071263', '17071264', '17071265', '17071266', '17071267', '17071268', '17071269', '17071270', '17071271', '17071272', '17071273', '17071274', '17071275', '17071276', '17071277', '17071278', '17071279', '17071280', '17071281', '17071284', '17071285', '17071286', '17071288', '17071289', '17071290', '17071291', '17071292', '17071293', '17071294', '17071295', '17071296', '17071297', '17071298', '17071299', '17071300', '17071301', '17071302', '17071303', '17071304', '17071305', '17071306', '17071307', '17071308', '17071309', '17071310', '17071311', '17071312', '17071313', '17071314', '17071315', '17071316', '17071317', '17071318', '17071319', '17071320', '17071321', '17071322', '17071323', '17071324', '17071325', '17071326', '17071328', '17071329', '17071330', '17071331', '17071334', '17071335', '17071336', '17071337', '17071338', '17071339', '17071340', '17071341', '17071342', '17071343', '17071344', '17071345', '17071346', '17071347', '17071348', '17071349', '17071350', '17071351', '17071352', '17071353', '17071355', '17071356', '17071357', '17071359', '17071360', '17071361', '17071362', '17071363', '17071365', '17071366', '17071367', '17071368', '17071369', '17071370', '17071372', '17071373', '17071374', '17071376', '17071377', '17071378', '17071379', '17071380', '17071381', '17071382', '17071383', '17071384', '17071385', '17071386', '17071387', '17071388', '17071389', '17071390', '17071391', '17071392', '17071393', '17071395', '17071396', '17071397', '17071398', '17071399', '17071401', '17071402', '17071403', '17071404', '17071405', '17071406', '17071407', '17071408', '17071409', '17071411', '17071413', '17071415', '17071416', '17071418', '17071419', '17071420', '17071421', '17071422', '17071423', '17071425', '17071426', '17071427', '17071428', '17071429', '17071430', '17071431', '17071433', '17071434', '17071435', '17071437', '17071445', '17071446', '17071447', '17071448', '17071449', '17071450', '17071451', '17071452', '17071124', '17071125', '17071126', '17071127', '17071128', '17071130', '17071131', '17071132', '17071133', '17071134', '17071135', '17071136', '17071137', '17071139', '17071141', '17071142', '17071143', '17071144', '17071148', '17071149', '17071151', '17071152', '17071153', '17071154', '17071156', '17071157', '17071158', '17071160', '17071162', '17071163', '17071209', '17071210', '17071211', '17071212', '17071213', '17071214', '17071215', '17071216', '17071217', '17071218', '17071219', '17071220', '17071221', '17071222', '17071224', '17071225', '17071226', '17071227', '17071228', '17071229', '17071164', '17071165', '17071166', '17071199', '17071208', '91474'] 2024-03-15 12:20:57 proc_lsst.multi:146 [INFO] found job 17071230 in provider checkpoint proc_lsst.multi INFO: found job 17071230 in provider checkpoint 2024-03-15 12:20:57 proc_lsst.multi:201 [INFO] cancelling 17071230 on provider checkpoint proc_lsst.multi INFO: cancelling 17071230 on provider checkpoint 2024-03-15 12:20:57 proc_lsst.multi:146 [INFO] found job 17071231 in provider checkpoint proc_lsst.multi INFO: found job 17071231 in provider checkpoint 2024-03-15 12:20:57 proc_lsst.multi:201 [INFO] cancelling 17071231 on provider checkpoint proc_lsst.multi INFO: cancelling 17071231 on provider checkpoint 2024-03-15 12:20:58 proc_lsst.multi:146 [INFO] found job 17071232 in provider checkpoint proc_lsst.multi INFO: found job 17071232 in provider checkpoint 2024-03-15 12:20:58 proc_lsst.multi:201 [INFO] cancelling 17071232 on provider checkpoint proc_lsst.multi INFO: cancelling 17071232 on provider checkpoint 2024-03-15 12:20:58 proc_lsst.multi:146 [INFO] found job 17071235 in provider checkpoint proc_lsst.multi INFO: found job 17071235 in provider checkpoint 2024-03-15 12:20:58 proc_lsst.multi:201 [INFO] cancelling 17071235 on provider checkpoint proc_lsst.multi INFO: cancelling 17071235 on provider checkpoint 2024-03-15 12:20:59 proc_lsst.multi:146 [INFO] found job 17071237 in provider checkpoint proc_lsst.multi INFO: found job 17071237 in provider checkpoint 2024-03-15 12:20:59 proc_lsst.multi:201 [INFO] cancelling 17071237 on provider checkpoint proc_lsst.multi INFO: cancelling 17071237 on provider checkpoint 2024-03-15 12:20:59 proc_lsst.multi:146 [INFO] found job 17071238 in provider checkpoint proc_lsst.multi INFO: found job 17071238 in provider checkpoint 2024-03-15 12:20:59 proc_lsst.multi:201 [INFO] cancelling 17071238 on provider checkpoint proc_lsst.multi INFO: cancelling 17071238 on provider checkpoint 2024-03-15 12:20:59 proc_lsst.multi:146 [INFO] found job 17071239 in provider checkpoint proc_lsst.multi INFO: found job 17071239 in provider checkpoint 2024-03-15 12:20:59 proc_lsst.multi:201 [INFO] cancelling 17071239 on provider checkpoint proc_lsst.multi INFO: cancelling 17071239 on provider checkpoint 2024-03-15 12:21:00 proc_lsst.multi:146 [INFO] found job 17071240 in provider checkpoint proc_lsst.multi INFO: found job 17071240 in provider checkpoint 2024-03-15 12:21:00 proc_lsst.multi:201 [INFO] cancelling 17071240 on provider checkpoint proc_lsst.multi INFO: cancelling 17071240 on provider checkpoint 2024-03-15 12:21:00 proc_lsst.multi:146 [INFO] found job 17071243 in provider checkpoint proc_lsst.multi INFO: found job 17071243 in provider checkpoint 2024-03-15 12:21:00 proc_lsst.multi:201 [INFO] cancelling 17071243 on provider checkpoint proc_lsst.multi INFO: cancelling 17071243 on provider checkpoint 2024-03-15 12:21:00 proc_lsst.multi:146 [INFO] found job 17071244 in provider checkpoint proc_lsst.multi INFO: found job 17071244 in provider checkpoint 2024-03-15 12:21:00 proc_lsst.multi:201 [INFO] cancelling 17071244 on provider checkpoint proc_lsst.multi INFO: cancelling 17071244 on provider checkpoint 2024-03-15 12:21:01 proc_lsst.multi:146 [INFO] found job 17071245 in provider checkpoint proc_lsst.multi INFO: found job 17071245 in provider checkpoint 2024-03-15 12:21:01 proc_lsst.multi:201 [INFO] cancelling 17071245 on provider checkpoint proc_lsst.multi INFO: cancelling 17071245 on provider checkpoint 2024-03-15 12:21:01 proc_lsst.multi:146 [INFO] found job 17071247 in provider checkpoint proc_lsst.multi INFO: found job 17071247 in provider checkpoint 2024-03-15 12:21:01 proc_lsst.multi:201 [INFO] cancelling 17071247 on provider checkpoint proc_lsst.multi INFO: cancelling 17071247 on provider checkpoint 2024-03-15 12:21:02 proc_lsst.multi:146 [INFO] found job 17071248 in provider checkpoint proc_lsst.multi INFO: found job 17071248 in provider checkpoint 2024-03-15 12:21:02 proc_lsst.multi:201 [INFO] cancelling 17071248 on provider checkpoint proc_lsst.multi INFO: cancelling 17071248 on provider checkpoint 2024-03-15 12:21:02 proc_lsst.multi:146 [INFO] found job 17071249 in provider checkpoint proc_lsst.multi INFO: found job 17071249 in provider checkpoint 2024-03-15 12:21:02 proc_lsst.multi:201 [INFO] cancelling 17071249 on provider checkpoint proc_lsst.multi INFO: cancelling 17071249 on provider checkpoint 2024-03-15 12:21:02 proc_lsst.multi:146 [INFO] found job 17071250 in provider checkpoint proc_lsst.multi INFO: found job 17071250 in provider checkpoint 2024-03-15 12:21:02 proc_lsst.multi:201 [INFO] cancelling 17071250 on provider checkpoint proc_lsst.multi INFO: cancelling 17071250 on provider checkpoint 2024-03-15 12:21:03 proc_lsst.multi:146 [INFO] found job 17071251 in provider checkpoint proc_lsst.multi INFO: found job 17071251 in provider checkpoint 2024-03-15 12:21:03 proc_lsst.multi:201 [INFO] cancelling 17071251 on provider checkpoint proc_lsst.multi INFO: cancelling 17071251 on provider checkpoint 2024-03-15 12:21:03 proc_lsst.multi:146 [INFO] found job 17071252 in provider checkpoint proc_lsst.multi INFO: found job 17071252 in provider checkpoint 2024-03-15 12:21:03 proc_lsst.multi:201 [INFO] cancelling 17071252 on provider checkpoint proc_lsst.multi INFO: cancelling 17071252 on provider checkpoint 2024-03-15 12:21:04 proc_lsst.multi:146 [INFO] found job 17071253 in provider checkpoint proc_lsst.multi INFO: found job 17071253 in provider checkpoint 2024-03-15 12:21:04 proc_lsst.multi:201 [INFO] cancelling 17071253 on provider checkpoint proc_lsst.multi INFO: cancelling 17071253 on provider checkpoint 2024-03-15 12:21:04 proc_lsst.multi:146 [INFO] found job 17071257 in provider checkpoint proc_lsst.multi INFO: found job 17071257 in provider checkpoint 2024-03-15 12:21:04 proc_lsst.multi:201 [INFO] cancelling 17071257 on provider checkpoint proc_lsst.multi INFO: cancelling 17071257 on provider checkpoint 2024-03-15 12:21:04 proc_lsst.multi:146 [INFO] found job 17071258 in provider checkpoint proc_lsst.multi INFO: found job 17071258 in provider checkpoint 2024-03-15 12:21:04 proc_lsst.multi:201 [INFO] cancelling 17071258 on provider checkpoint proc_lsst.multi INFO: cancelling 17071258 on provider checkpoint 2024-03-15 12:21:05 proc_lsst.multi:146 [INFO] found job 17071259 in provider checkpoint proc_lsst.multi INFO: found job 17071259 in provider checkpoint 2024-03-15 12:21:05 proc_lsst.multi:201 [INFO] cancelling 17071259 on provider checkpoint proc_lsst.multi INFO: cancelling 17071259 on provider checkpoint 2024-03-15 12:21:05 proc_lsst.multi:146 [INFO] found job 17071261 in provider checkpoint proc_lsst.multi INFO: found job 17071261 in provider checkpoint 2024-03-15 12:21:05 proc_lsst.multi:201 [INFO] cancelling 17071261 on provider checkpoint proc_lsst.multi INFO: cancelling 17071261 on provider checkpoint 2024-03-15 12:21:06 proc_lsst.multi:146 [INFO] found job 17071263 in provider checkpoint proc_lsst.multi INFO: found job 17071263 in provider checkpoint 2024-03-15 12:21:06 proc_lsst.multi:201 [INFO] cancelling 17071263 on provider checkpoint proc_lsst.multi INFO: cancelling 17071263 on provider checkpoint 2024-03-15 12:21:06 proc_lsst.multi:146 [INFO] found job 17071264 in provider checkpoint proc_lsst.multi INFO: found job 17071264 in provider checkpoint 2024-03-15 12:21:06 proc_lsst.multi:201 [INFO] cancelling 17071264 on provider checkpoint proc_lsst.multi INFO: cancelling 17071264 on provider checkpoint 2024-03-15 12:21:06 proc_lsst.multi:146 [INFO] found job 17071265 in provider checkpoint proc_lsst.multi INFO: found job 17071265 in provider checkpoint 2024-03-15 12:21:06 proc_lsst.multi:201 [INFO] cancelling 17071265 on provider checkpoint proc_lsst.multi INFO: cancelling 17071265 on provider checkpoint 2024-03-15 12:21:07 proc_lsst.multi:146 [INFO] found job 17071266 in provider checkpoint proc_lsst.multi INFO: found job 17071266 in provider checkpoint 2024-03-15 12:21:07 proc_lsst.multi:201 [INFO] cancelling 17071266 on provider checkpoint proc_lsst.multi INFO: cancelling 17071266 on provider checkpoint 2024-03-15 12:21:07 proc_lsst.multi:146 [INFO] found job 17071267 in provider checkpoint proc_lsst.multi INFO: found job 17071267 in provider checkpoint 2024-03-15 12:21:07 proc_lsst.multi:201 [INFO] cancelling 17071267 on provider checkpoint proc_lsst.multi INFO: cancelling 17071267 on provider checkpoint 2024-03-15 12:21:09 proc_lsst.multi:146 [INFO] found job 17071268 in provider checkpoint proc_lsst.multi INFO: found job 17071268 in provider checkpoint 2024-03-15 12:21:09 proc_lsst.multi:201 [INFO] cancelling 17071268 on provider checkpoint proc_lsst.multi INFO: cancelling 17071268 on provider checkpoint 2024-03-15 12:21:10 proc_lsst.multi:146 [INFO] found job 17071269 in provider checkpoint proc_lsst.multi INFO: found job 17071269 in provider checkpoint 2024-03-15 12:21:10 proc_lsst.multi:201 [INFO] cancelling 17071269 on provider checkpoint proc_lsst.multi INFO: cancelling 17071269 on provider checkpoint 2024-03-15 12:21:11 proc_lsst.multi:146 [INFO] found job 17071270 in provider checkpoint proc_lsst.multi INFO: found job 17071270 in provider checkpoint 2024-03-15 12:21:11 proc_lsst.multi:201 [INFO] cancelling 17071270 on provider checkpoint proc_lsst.multi INFO: cancelling 17071270 on provider checkpoint 2024-03-15 12:21:11 proc_lsst.multi:146 [INFO] found job 17071271 in provider checkpoint proc_lsst.multi INFO: found job 17071271 in provider checkpoint 2024-03-15 12:21:11 proc_lsst.multi:201 [INFO] cancelling 17071271 on provider checkpoint proc_lsst.multi INFO: cancelling 17071271 on provider checkpoint 2024-03-15 12:21:11 proc_lsst.multi:146 [INFO] found job 17071272 in provider checkpoint proc_lsst.multi INFO: found job 17071272 in provider checkpoint 2024-03-15 12:21:11 proc_lsst.multi:201 [INFO] cancelling 17071272 on provider checkpoint proc_lsst.multi INFO: cancelling 17071272 on provider checkpoint 2024-03-15 12:21:12 proc_lsst.multi:146 [INFO] found job 17071273 in provider checkpoint proc_lsst.multi INFO: found job 17071273 in provider checkpoint 2024-03-15 12:21:12 proc_lsst.multi:201 [INFO] cancelling 17071273 on provider checkpoint proc_lsst.multi INFO: cancelling 17071273 on provider checkpoint 2024-03-15 12:21:12 proc_lsst.multi:146 [INFO] found job 17071274 in provider checkpoint proc_lsst.multi INFO: found job 17071274 in provider checkpoint 2024-03-15 12:21:12 proc_lsst.multi:201 [INFO] cancelling 17071274 on provider checkpoint proc_lsst.multi INFO: cancelling 17071274 on provider checkpoint 2024-03-15 12:21:13 proc_lsst.multi:146 [INFO] found job 17071275 in provider checkpoint proc_lsst.multi INFO: found job 17071275 in provider checkpoint 2024-03-15 12:21:13 proc_lsst.multi:201 [INFO] cancelling 17071275 on provider checkpoint proc_lsst.multi INFO: cancelling 17071275 on provider checkpoint 2024-03-15 12:21:13 proc_lsst.multi:146 [INFO] found job 17071276 in provider checkpoint proc_lsst.multi INFO: found job 17071276 in provider checkpoint 2024-03-15 12:21:13 proc_lsst.multi:201 [INFO] cancelling 17071276 on provider checkpoint proc_lsst.multi INFO: cancelling 17071276 on provider checkpoint 2024-03-15 12:21:13 proc_lsst.multi:146 [INFO] found job 17071277 in provider checkpoint proc_lsst.multi INFO: found job 17071277 in provider checkpoint 2024-03-15 12:21:13 proc_lsst.multi:201 [INFO] cancelling 17071277 on provider checkpoint proc_lsst.multi INFO: cancelling 17071277 on provider checkpoint 2024-03-15 12:21:14 proc_lsst.multi:146 [INFO] found job 17071278 in provider checkpoint proc_lsst.multi INFO: found job 17071278 in provider checkpoint 2024-03-15 12:21:14 proc_lsst.multi:201 [INFO] cancelling 17071278 on provider checkpoint proc_lsst.multi INFO: cancelling 17071278 on provider checkpoint 2024-03-15 12:21:14 proc_lsst.multi:146 [INFO] found job 17071279 in provider checkpoint proc_lsst.multi INFO: found job 17071279 in provider checkpoint 2024-03-15 12:21:14 proc_lsst.multi:201 [INFO] cancelling 17071279 on provider checkpoint proc_lsst.multi INFO: cancelling 17071279 on provider checkpoint 2024-03-15 12:21:15 proc_lsst.multi:146 [INFO] found job 17071280 in provider checkpoint proc_lsst.multi INFO: found job 17071280 in provider checkpoint 2024-03-15 12:21:15 proc_lsst.multi:201 [INFO] cancelling 17071280 on provider checkpoint proc_lsst.multi INFO: cancelling 17071280 on provider checkpoint 2024-03-15 12:21:15 proc_lsst.multi:146 [INFO] found job 17071281 in provider checkpoint proc_lsst.multi INFO: found job 17071281 in provider checkpoint 2024-03-15 12:21:15 proc_lsst.multi:201 [INFO] cancelling 17071281 on provider checkpoint proc_lsst.multi INFO: cancelling 17071281 on provider checkpoint 2024-03-15 12:21:15 proc_lsst.multi:146 [INFO] found job 17071284 in provider checkpoint proc_lsst.multi INFO: found job 17071284 in provider checkpoint 2024-03-15 12:21:15 proc_lsst.multi:201 [INFO] cancelling 17071284 on provider checkpoint proc_lsst.multi INFO: cancelling 17071284 on provider checkpoint 2024-03-15 12:21:16 proc_lsst.multi:146 [INFO] found job 17071285 in provider checkpoint proc_lsst.multi INFO: found job 17071285 in provider checkpoint 2024-03-15 12:21:16 proc_lsst.multi:201 [INFO] cancelling 17071285 on provider checkpoint proc_lsst.multi INFO: cancelling 17071285 on provider checkpoint 2024-03-15 12:21:16 proc_lsst.multi:146 [INFO] found job 17071286 in provider checkpoint proc_lsst.multi INFO: found job 17071286 in provider checkpoint 2024-03-15 12:21:16 proc_lsst.multi:201 [INFO] cancelling 17071286 on provider checkpoint proc_lsst.multi INFO: cancelling 17071286 on provider checkpoint 2024-03-15 12:21:17 proc_lsst.multi:146 [INFO] found job 17071288 in provider checkpoint proc_lsst.multi INFO: found job 17071288 in provider checkpoint 2024-03-15 12:21:17 proc_lsst.multi:201 [INFO] cancelling 17071288 on provider checkpoint proc_lsst.multi INFO: cancelling 17071288 on provider checkpoint 2024-03-15 12:21:17 proc_lsst.multi:146 [INFO] found job 17071289 in provider checkpoint proc_lsst.multi INFO: found job 17071289 in provider checkpoint 2024-03-15 12:21:17 proc_lsst.multi:201 [INFO] cancelling 17071289 on provider checkpoint proc_lsst.multi INFO: cancelling 17071289 on provider checkpoint 2024-03-15 12:21:17 proc_lsst.multi:146 [INFO] found job 17071290 in provider checkpoint proc_lsst.multi INFO: found job 17071290 in provider checkpoint 2024-03-15 12:21:17 proc_lsst.multi:201 [INFO] cancelling 17071290 on provider checkpoint proc_lsst.multi INFO: cancelling 17071290 on provider checkpoint 2024-03-15 12:21:18 proc_lsst.multi:146 [INFO] found job 17071291 in provider checkpoint proc_lsst.multi INFO: found job 17071291 in provider checkpoint 2024-03-15 12:21:18 proc_lsst.multi:201 [INFO] cancelling 17071291 on provider checkpoint proc_lsst.multi INFO: cancelling 17071291 on provider checkpoint 2024-03-15 12:21:18 proc_lsst.multi:146 [INFO] found job 17071292 in provider checkpoint proc_lsst.multi INFO: found job 17071292 in provider checkpoint 2024-03-15 12:21:18 proc_lsst.multi:201 [INFO] cancelling 17071292 on provider checkpoint proc_lsst.multi INFO: cancelling 17071292 on provider checkpoint 2024-03-15 12:21:18 proc_lsst.multi:146 [INFO] found job 17071293 in provider checkpoint proc_lsst.multi INFO: found job 17071293 in provider checkpoint 2024-03-15 12:21:18 proc_lsst.multi:201 [INFO] cancelling 17071293 on provider checkpoint proc_lsst.multi INFO: cancelling 17071293 on provider checkpoint 2024-03-15 12:21:19 proc_lsst.multi:146 [INFO] found job 17071294 in provider checkpoint proc_lsst.multi INFO: found job 17071294 in provider checkpoint 2024-03-15 12:21:19 proc_lsst.multi:201 [INFO] cancelling 17071294 on provider checkpoint proc_lsst.multi INFO: cancelling 17071294 on provider checkpoint 2024-03-15 12:21:19 proc_lsst.multi:146 [INFO] found job 17071295 in provider checkpoint proc_lsst.multi INFO: found job 17071295 in provider checkpoint 2024-03-15 12:21:19 proc_lsst.multi:201 [INFO] cancelling 17071295 on provider checkpoint proc_lsst.multi INFO: cancelling 17071295 on provider checkpoint 2024-03-15 12:21:20 proc_lsst.multi:146 [INFO] found job 17071296 in provider checkpoint proc_lsst.multi INFO: found job 17071296 in provider checkpoint 2024-03-15 12:21:20 proc_lsst.multi:201 [INFO] cancelling 17071296 on provider checkpoint proc_lsst.multi INFO: cancelling 17071296 on provider checkpoint 2024-03-15 12:21:20 proc_lsst.multi:146 [INFO] found job 17071297 in provider checkpoint proc_lsst.multi INFO: found job 17071297 in provider checkpoint 2024-03-15 12:21:20 proc_lsst.multi:201 [INFO] cancelling 17071297 on provider checkpoint proc_lsst.multi INFO: cancelling 17071297 on provider checkpoint 2024-03-15 12:21:20 proc_lsst.multi:146 [INFO] found job 17071298 in provider checkpoint proc_lsst.multi INFO: found job 17071298 in provider checkpoint 2024-03-15 12:21:20 proc_lsst.multi:201 [INFO] cancelling 17071298 on provider checkpoint proc_lsst.multi INFO: cancelling 17071298 on provider checkpoint 2024-03-15 12:21:21 proc_lsst.multi:146 [INFO] found job 17071299 in provider checkpoint proc_lsst.multi INFO: found job 17071299 in provider checkpoint 2024-03-15 12:21:21 proc_lsst.multi:201 [INFO] cancelling 17071299 on provider checkpoint proc_lsst.multi INFO: cancelling 17071299 on provider checkpoint 2024-03-15 12:21:21 proc_lsst.multi:146 [INFO] found job 17071300 in provider checkpoint proc_lsst.multi INFO: found job 17071300 in provider checkpoint 2024-03-15 12:21:21 proc_lsst.multi:201 [INFO] cancelling 17071300 on provider checkpoint proc_lsst.multi INFO: cancelling 17071300 on provider checkpoint 2024-03-15 12:21:22 proc_lsst.multi:146 [INFO] found job 17071301 in provider checkpoint proc_lsst.multi INFO: found job 17071301 in provider checkpoint 2024-03-15 12:21:22 proc_lsst.multi:201 [INFO] cancelling 17071301 on provider checkpoint proc_lsst.multi INFO: cancelling 17071301 on provider checkpoint 2024-03-15 12:21:22 proc_lsst.multi:146 [INFO] found job 17071302 in provider checkpoint proc_lsst.multi INFO: found job 17071302 in provider checkpoint 2024-03-15 12:21:22 proc_lsst.multi:201 [INFO] cancelling 17071302 on provider checkpoint proc_lsst.multi INFO: cancelling 17071302 on provider checkpoint 2024-03-15 12:21:22 proc_lsst.multi:146 [INFO] found job 17071303 in provider checkpoint proc_lsst.multi INFO: found job 17071303 in provider checkpoint 2024-03-15 12:21:22 proc_lsst.multi:201 [INFO] cancelling 17071303 on provider checkpoint proc_lsst.multi INFO: cancelling 17071303 on provider checkpoint 2024-03-15 12:21:23 proc_lsst.multi:146 [INFO] found job 17071304 in provider checkpoint proc_lsst.multi INFO: found job 17071304 in provider checkpoint 2024-03-15 12:21:23 proc_lsst.multi:201 [INFO] cancelling 17071304 on provider checkpoint proc_lsst.multi INFO: cancelling 17071304 on provider checkpoint 2024-03-15 12:21:23 proc_lsst.multi:146 [INFO] found job 17071305 in provider checkpoint proc_lsst.multi INFO: found job 17071305 in provider checkpoint 2024-03-15 12:21:23 proc_lsst.multi:201 [INFO] cancelling 17071305 on provider checkpoint proc_lsst.multi INFO: cancelling 17071305 on provider checkpoint 2024-03-15 12:21:24 proc_lsst.multi:146 [INFO] found job 17071306 in provider checkpoint proc_lsst.multi INFO: found job 17071306 in provider checkpoint 2024-03-15 12:21:24 proc_lsst.multi:201 [INFO] cancelling 17071306 on provider checkpoint proc_lsst.multi INFO: cancelling 17071306 on provider checkpoint 2024-03-15 12:21:24 proc_lsst.multi:146 [INFO] found job 17071307 in provider checkpoint proc_lsst.multi INFO: found job 17071307 in provider checkpoint 2024-03-15 12:21:24 proc_lsst.multi:201 [INFO] cancelling 17071307 on provider checkpoint proc_lsst.multi INFO: cancelling 17071307 on provider checkpoint 2024-03-15 12:21:25 proc_lsst.multi:146 [INFO] found job 17071308 in provider checkpoint proc_lsst.multi INFO: found job 17071308 in provider checkpoint 2024-03-15 12:21:25 proc_lsst.multi:201 [INFO] cancelling 17071308 on provider checkpoint proc_lsst.multi INFO: cancelling 17071308 on provider checkpoint 2024-03-15 12:21:25 proc_lsst.multi:146 [INFO] found job 17071309 in provider checkpoint proc_lsst.multi INFO: found job 17071309 in provider checkpoint 2024-03-15 12:21:25 proc_lsst.multi:201 [INFO] cancelling 17071309 on provider checkpoint proc_lsst.multi INFO: cancelling 17071309 on provider checkpoint 2024-03-15 12:21:25 proc_lsst.multi:146 [INFO] found job 17071310 in provider checkpoint proc_lsst.multi INFO: found job 17071310 in provider checkpoint 2024-03-15 12:21:25 proc_lsst.multi:201 [INFO] cancelling 17071310 on provider checkpoint proc_lsst.multi INFO: cancelling 17071310 on provider checkpoint 2024-03-15 12:21:26 proc_lsst.multi:146 [INFO] found job 17071311 in provider checkpoint proc_lsst.multi INFO: found job 17071311 in provider checkpoint 2024-03-15 12:21:26 proc_lsst.multi:201 [INFO] cancelling 17071311 on provider checkpoint proc_lsst.multi INFO: cancelling 17071311 on provider checkpoint 2024-03-15 12:21:26 proc_lsst.multi:146 [INFO] found job 17071312 in provider checkpoint proc_lsst.multi INFO: found job 17071312 in provider checkpoint 2024-03-15 12:21:26 proc_lsst.multi:201 [INFO] cancelling 17071312 on provider checkpoint proc_lsst.multi INFO: cancelling 17071312 on provider checkpoint 2024-03-15 12:21:27 proc_lsst.multi:146 [INFO] found job 17071313 in provider checkpoint proc_lsst.multi INFO: found job 17071313 in provider checkpoint 2024-03-15 12:21:27 proc_lsst.multi:201 [INFO] cancelling 17071313 on provider checkpoint proc_lsst.multi INFO: cancelling 17071313 on provider checkpoint 2024-03-15 12:21:27 proc_lsst.multi:146 [INFO] found job 17071314 in provider checkpoint proc_lsst.multi INFO: found job 17071314 in provider checkpoint 2024-03-15 12:21:27 proc_lsst.multi:201 [INFO] cancelling 17071314 on provider checkpoint proc_lsst.multi INFO: cancelling 17071314 on provider checkpoint 2024-03-15 12:21:27 proc_lsst.multi:146 [INFO] found job 17071315 in provider checkpoint proc_lsst.multi INFO: found job 17071315 in provider checkpoint 2024-03-15 12:21:27 proc_lsst.multi:201 [INFO] cancelling 17071315 on provider checkpoint proc_lsst.multi INFO: cancelling 17071315 on provider checkpoint 2024-03-15 12:21:28 proc_lsst.multi:146 [INFO] found job 17071316 in provider checkpoint proc_lsst.multi INFO: found job 17071316 in provider checkpoint 2024-03-15 12:21:28 proc_lsst.multi:201 [INFO] cancelling 17071316 on provider checkpoint proc_lsst.multi INFO: cancelling 17071316 on provider checkpoint 2024-03-15 12:21:28 proc_lsst.multi:146 [INFO] found job 17071317 in provider checkpoint proc_lsst.multi INFO: found job 17071317 in provider checkpoint 2024-03-15 12:21:28 proc_lsst.multi:201 [INFO] cancelling 17071317 on provider checkpoint proc_lsst.multi INFO: cancelling 17071317 on provider checkpoint 2024-03-15 12:21:29 proc_lsst.multi:146 [INFO] found job 17071318 in provider checkpoint proc_lsst.multi INFO: found job 17071318 in provider checkpoint 2024-03-15 12:21:29 proc_lsst.multi:201 [INFO] cancelling 17071318 on provider checkpoint proc_lsst.multi INFO: cancelling 17071318 on provider checkpoint 2024-03-15 12:21:29 proc_lsst.multi:146 [INFO] found job 17071319 in provider checkpoint proc_lsst.multi INFO: found job 17071319 in provider checkpoint 2024-03-15 12:21:29 proc_lsst.multi:201 [INFO] cancelling 17071319 on provider checkpoint proc_lsst.multi INFO: cancelling 17071319 on provider checkpoint 2024-03-15 12:21:29 proc_lsst.multi:146 [INFO] found job 17071320 in provider checkpoint proc_lsst.multi INFO: found job 17071320 in provider checkpoint 2024-03-15 12:21:29 proc_lsst.multi:201 [INFO] cancelling 17071320 on provider checkpoint proc_lsst.multi INFO: cancelling 17071320 on provider checkpoint 2024-03-15 12:21:30 proc_lsst.multi:146 [INFO] found job 17071321 in provider checkpoint proc_lsst.multi INFO: found job 17071321 in provider checkpoint 2024-03-15 12:21:30 proc_lsst.multi:201 [INFO] cancelling 17071321 on provider checkpoint proc_lsst.multi INFO: cancelling 17071321 on provider checkpoint 2024-03-15 12:21:30 proc_lsst.multi:146 [INFO] found job 17071322 in provider checkpoint proc_lsst.multi INFO: found job 17071322 in provider checkpoint 2024-03-15 12:21:30 proc_lsst.multi:201 [INFO] cancelling 17071322 on provider checkpoint proc_lsst.multi INFO: cancelling 17071322 on provider checkpoint 2024-03-15 12:21:31 proc_lsst.multi:146 [INFO] found job 17071323 in provider checkpoint proc_lsst.multi INFO: found job 17071323 in provider checkpoint 2024-03-15 12:21:31 proc_lsst.multi:201 [INFO] cancelling 17071323 on provider checkpoint proc_lsst.multi INFO: cancelling 17071323 on provider checkpoint 2024-03-15 12:21:31 proc_lsst.multi:146 [INFO] found job 17071324 in provider checkpoint proc_lsst.multi INFO: found job 17071324 in provider checkpoint 2024-03-15 12:21:31 proc_lsst.multi:201 [INFO] cancelling 17071324 on provider checkpoint proc_lsst.multi INFO: cancelling 17071324 on provider checkpoint 2024-03-15 12:21:31 proc_lsst.multi:146 [INFO] found job 17071325 in provider checkpoint proc_lsst.multi INFO: found job 17071325 in provider checkpoint 2024-03-15 12:21:31 proc_lsst.multi:201 [INFO] cancelling 17071325 on provider checkpoint proc_lsst.multi INFO: cancelling 17071325 on provider checkpoint 2024-03-15 12:21:32 proc_lsst.multi:146 [INFO] found job 17071326 in provider checkpoint proc_lsst.multi INFO: found job 17071326 in provider checkpoint 2024-03-15 12:21:32 proc_lsst.multi:201 [INFO] cancelling 17071326 on provider checkpoint proc_lsst.multi INFO: cancelling 17071326 on provider checkpoint 2024-03-15 12:21:32 proc_lsst.multi:146 [INFO] found job 17071328 in provider checkpoint proc_lsst.multi INFO: found job 17071328 in provider checkpoint 2024-03-15 12:21:32 proc_lsst.multi:201 [INFO] cancelling 17071328 on provider checkpoint proc_lsst.multi INFO: cancelling 17071328 on provider checkpoint 2024-03-15 12:21:32 proc_lsst.multi:146 [INFO] found job 17071329 in provider checkpoint proc_lsst.multi INFO: found job 17071329 in provider checkpoint 2024-03-15 12:21:32 proc_lsst.multi:201 [INFO] cancelling 17071329 on provider checkpoint proc_lsst.multi INFO: cancelling 17071329 on provider checkpoint 2024-03-15 12:21:33 proc_lsst.multi:146 [INFO] found job 17071330 in provider checkpoint proc_lsst.multi INFO: found job 17071330 in provider checkpoint 2024-03-15 12:21:33 proc_lsst.multi:201 [INFO] cancelling 17071330 on provider checkpoint proc_lsst.multi INFO: cancelling 17071330 on provider checkpoint 2024-03-15 12:21:33 proc_lsst.multi:146 [INFO] found job 17071331 in provider checkpoint proc_lsst.multi INFO: found job 17071331 in provider checkpoint 2024-03-15 12:21:33 proc_lsst.multi:201 [INFO] cancelling 17071331 on provider checkpoint proc_lsst.multi INFO: cancelling 17071331 on provider checkpoint 2024-03-15 12:21:34 proc_lsst.multi:146 [INFO] found job 17071334 in provider checkpoint proc_lsst.multi INFO: found job 17071334 in provider checkpoint 2024-03-15 12:21:34 proc_lsst.multi:201 [INFO] cancelling 17071334 on provider checkpoint proc_lsst.multi INFO: cancelling 17071334 on provider checkpoint 2024-03-15 12:21:34 proc_lsst.multi:146 [INFO] found job 17071335 in provider checkpoint proc_lsst.multi INFO: found job 17071335 in provider checkpoint 2024-03-15 12:21:34 proc_lsst.multi:201 [INFO] cancelling 17071335 on provider checkpoint proc_lsst.multi INFO: cancelling 17071335 on provider checkpoint 2024-03-15 12:21:34 proc_lsst.multi:146 [INFO] found job 17071336 in provider checkpoint proc_lsst.multi INFO: found job 17071336 in provider checkpoint 2024-03-15 12:21:34 proc_lsst.multi:201 [INFO] cancelling 17071336 on provider checkpoint proc_lsst.multi INFO: cancelling 17071336 on provider checkpoint 2024-03-15 12:21:35 proc_lsst.multi:146 [INFO] found job 17071337 in provider checkpoint proc_lsst.multi INFO: found job 17071337 in provider checkpoint 2024-03-15 12:21:35 proc_lsst.multi:201 [INFO] cancelling 17071337 on provider checkpoint proc_lsst.multi INFO: cancelling 17071337 on provider checkpoint 2024-03-15 12:21:35 proc_lsst.multi:146 [INFO] found job 17071338 in provider checkpoint proc_lsst.multi INFO: found job 17071338 in provider checkpoint 2024-03-15 12:21:35 proc_lsst.multi:201 [INFO] cancelling 17071338 on provider checkpoint proc_lsst.multi INFO: cancelling 17071338 on provider checkpoint 2024-03-15 12:21:35 proc_lsst.multi:146 [INFO] found job 17071339 in provider checkpoint proc_lsst.multi INFO: found job 17071339 in provider checkpoint 2024-03-15 12:21:35 proc_lsst.multi:201 [INFO] cancelling 17071339 on provider checkpoint proc_lsst.multi INFO: cancelling 17071339 on provider checkpoint 2024-03-15 12:21:35 proc_lsst.multi:146 [INFO] found job 17071340 in provider checkpoint proc_lsst.multi INFO: found job 17071340 in provider checkpoint 2024-03-15 12:21:35 proc_lsst.multi:201 [INFO] cancelling 17071340 on provider checkpoint proc_lsst.multi INFO: cancelling 17071340 on provider checkpoint 2024-03-15 12:21:36 proc_lsst.multi:146 [INFO] found job 17071341 in provider checkpoint proc_lsst.multi INFO: found job 17071341 in provider checkpoint 2024-03-15 12:21:36 proc_lsst.multi:201 [INFO] cancelling 17071341 on provider checkpoint proc_lsst.multi INFO: cancelling 17071341 on provider checkpoint 2024-03-15 12:21:36 proc_lsst.multi:146 [INFO] found job 17071342 in provider checkpoint proc_lsst.multi INFO: found job 17071342 in provider checkpoint 2024-03-15 12:21:36 proc_lsst.multi:201 [INFO] cancelling 17071342 on provider checkpoint proc_lsst.multi INFO: cancelling 17071342 on provider checkpoint 2024-03-15 12:21:37 proc_lsst.multi:146 [INFO] found job 17071343 in provider checkpoint proc_lsst.multi INFO: found job 17071343 in provider checkpoint 2024-03-15 12:21:37 proc_lsst.multi:201 [INFO] cancelling 17071343 on provider checkpoint proc_lsst.multi INFO: cancelling 17071343 on provider checkpoint 2024-03-15 12:21:37 proc_lsst.multi:146 [INFO] found job 17071344 in provider checkpoint proc_lsst.multi INFO: found job 17071344 in provider checkpoint 2024-03-15 12:21:37 proc_lsst.multi:201 [INFO] cancelling 17071344 on provider checkpoint proc_lsst.multi INFO: cancelling 17071344 on provider checkpoint 2024-03-15 12:21:37 proc_lsst.multi:146 [INFO] found job 17071345 in provider checkpoint proc_lsst.multi INFO: found job 17071345 in provider checkpoint 2024-03-15 12:21:37 proc_lsst.multi:201 [INFO] cancelling 17071345 on provider checkpoint proc_lsst.multi INFO: cancelling 17071345 on provider checkpoint 2024-03-15 12:21:38 proc_lsst.multi:146 [INFO] found job 17071346 in provider checkpoint proc_lsst.multi INFO: found job 17071346 in provider checkpoint 2024-03-15 12:21:38 proc_lsst.multi:201 [INFO] cancelling 17071346 on provider checkpoint proc_lsst.multi INFO: cancelling 17071346 on provider checkpoint 2024-03-15 12:21:38 proc_lsst.multi:146 [INFO] found job 17071347 in provider checkpoint proc_lsst.multi INFO: found job 17071347 in provider checkpoint 2024-03-15 12:21:38 proc_lsst.multi:201 [INFO] cancelling 17071347 on provider checkpoint proc_lsst.multi INFO: cancelling 17071347 on provider checkpoint 2024-03-15 12:21:39 proc_lsst.multi:146 [INFO] found job 17071348 in provider checkpoint proc_lsst.multi INFO: found job 17071348 in provider checkpoint 2024-03-15 12:21:39 proc_lsst.multi:201 [INFO] cancelling 17071348 on provider checkpoint proc_lsst.multi INFO: cancelling 17071348 on provider checkpoint 2024-03-15 12:21:39 proc_lsst.multi:146 [INFO] found job 17071349 in provider checkpoint proc_lsst.multi INFO: found job 17071349 in provider checkpoint 2024-03-15 12:21:39 proc_lsst.multi:201 [INFO] cancelling 17071349 on provider checkpoint proc_lsst.multi INFO: cancelling 17071349 on provider checkpoint 2024-03-15 12:21:39 proc_lsst.multi:146 [INFO] found job 17071350 in provider checkpoint proc_lsst.multi INFO: found job 17071350 in provider checkpoint 2024-03-15 12:21:39 proc_lsst.multi:201 [INFO] cancelling 17071350 on provider checkpoint proc_lsst.multi INFO: cancelling 17071350 on provider checkpoint 2024-03-15 12:21:40 proc_lsst.multi:146 [INFO] found job 17071351 in provider checkpoint proc_lsst.multi INFO: found job 17071351 in provider checkpoint 2024-03-15 12:21:40 proc_lsst.multi:201 [INFO] cancelling 17071351 on provider checkpoint proc_lsst.multi INFO: cancelling 17071351 on provider checkpoint 2024-03-15 12:21:40 proc_lsst.multi:146 [INFO] found job 17071352 in provider checkpoint proc_lsst.multi INFO: found job 17071352 in provider checkpoint 2024-03-15 12:21:40 proc_lsst.multi:201 [INFO] cancelling 17071352 on provider checkpoint proc_lsst.multi INFO: cancelling 17071352 on provider checkpoint 2024-03-15 12:21:40 proc_lsst.multi:146 [INFO] found job 17071353 in provider checkpoint proc_lsst.multi INFO: found job 17071353 in provider checkpoint 2024-03-15 12:21:40 proc_lsst.multi:201 [INFO] cancelling 17071353 on provider checkpoint proc_lsst.multi INFO: cancelling 17071353 on provider checkpoint 2024-03-15 12:21:42 proc_lsst.multi:146 [INFO] found job 17071355 in provider checkpoint proc_lsst.multi INFO: found job 17071355 in provider checkpoint 2024-03-15 12:21:42 proc_lsst.multi:201 [INFO] cancelling 17071355 on provider checkpoint proc_lsst.multi INFO: cancelling 17071355 on provider checkpoint 2024-03-15 12:21:43 proc_lsst.multi:146 [INFO] found job 17071356 in provider checkpoint proc_lsst.multi INFO: found job 17071356 in provider checkpoint 2024-03-15 12:21:43 proc_lsst.multi:201 [INFO] cancelling 17071356 on provider checkpoint proc_lsst.multi INFO: cancelling 17071356 on provider checkpoint 2024-03-15 12:21:43 proc_lsst.multi:146 [INFO] found job 17071357 in provider checkpoint proc_lsst.multi INFO: found job 17071357 in provider checkpoint 2024-03-15 12:21:43 proc_lsst.multi:201 [INFO] cancelling 17071357 on provider checkpoint proc_lsst.multi INFO: cancelling 17071357 on provider checkpoint 2024-03-15 12:21:44 proc_lsst.multi:146 [INFO] found job 17071359 in provider checkpoint proc_lsst.multi INFO: found job 17071359 in provider checkpoint 2024-03-15 12:21:44 proc_lsst.multi:201 [INFO] cancelling 17071359 on provider checkpoint proc_lsst.multi INFO: cancelling 17071359 on provider checkpoint 2024-03-15 12:21:44 proc_lsst.multi:146 [INFO] found job 17071360 in provider checkpoint proc_lsst.multi INFO: found job 17071360 in provider checkpoint 2024-03-15 12:21:44 proc_lsst.multi:201 [INFO] cancelling 17071360 on provider checkpoint proc_lsst.multi INFO: cancelling 17071360 on provider checkpoint 2024-03-15 12:21:44 proc_lsst.multi:146 [INFO] found job 17071361 in provider checkpoint proc_lsst.multi INFO: found job 17071361 in provider checkpoint 2024-03-15 12:21:44 proc_lsst.multi:201 [INFO] cancelling 17071361 on provider checkpoint proc_lsst.multi INFO: cancelling 17071361 on provider checkpoint 2024-03-15 12:21:45 proc_lsst.multi:146 [INFO] found job 17071362 in provider checkpoint proc_lsst.multi INFO: found job 17071362 in provider checkpoint 2024-03-15 12:21:45 proc_lsst.multi:201 [INFO] cancelling 17071362 on provider checkpoint proc_lsst.multi INFO: cancelling 17071362 on provider checkpoint 2024-03-15 12:21:45 proc_lsst.multi:146 [INFO] found job 17071363 in provider checkpoint proc_lsst.multi INFO: found job 17071363 in provider checkpoint 2024-03-15 12:21:45 proc_lsst.multi:201 [INFO] cancelling 17071363 on provider checkpoint proc_lsst.multi INFO: cancelling 17071363 on provider checkpoint 2024-03-15 12:21:45 proc_lsst.multi:146 [INFO] found job 17071365 in provider checkpoint proc_lsst.multi INFO: found job 17071365 in provider checkpoint 2024-03-15 12:21:45 proc_lsst.multi:201 [INFO] cancelling 17071365 on provider checkpoint proc_lsst.multi INFO: cancelling 17071365 on provider checkpoint 2024-03-15 12:21:46 proc_lsst.multi:146 [INFO] found job 17071366 in provider checkpoint proc_lsst.multi INFO: found job 17071366 in provider checkpoint 2024-03-15 12:21:46 proc_lsst.multi:201 [INFO] cancelling 17071366 on provider checkpoint proc_lsst.multi INFO: cancelling 17071366 on provider checkpoint 2024-03-15 12:21:46 proc_lsst.multi:146 [INFO] found job 17071367 in provider checkpoint proc_lsst.multi INFO: found job 17071367 in provider checkpoint 2024-03-15 12:21:46 proc_lsst.multi:201 [INFO] cancelling 17071367 on provider checkpoint proc_lsst.multi INFO: cancelling 17071367 on provider checkpoint 2024-03-15 12:21:47 proc_lsst.multi:146 [INFO] found job 17071368 in provider checkpoint proc_lsst.multi INFO: found job 17071368 in provider checkpoint 2024-03-15 12:21:47 proc_lsst.multi:201 [INFO] cancelling 17071368 on provider checkpoint proc_lsst.multi INFO: cancelling 17071368 on provider checkpoint 2024-03-15 12:21:47 proc_lsst.multi:146 [INFO] found job 17071369 in provider checkpoint proc_lsst.multi INFO: found job 17071369 in provider checkpoint 2024-03-15 12:21:47 proc_lsst.multi:201 [INFO] cancelling 17071369 on provider checkpoint proc_lsst.multi INFO: cancelling 17071369 on provider checkpoint 2024-03-15 12:21:47 proc_lsst.multi:146 [INFO] found job 17071370 in provider checkpoint proc_lsst.multi INFO: found job 17071370 in provider checkpoint 2024-03-15 12:21:47 proc_lsst.multi:201 [INFO] cancelling 17071370 on provider checkpoint proc_lsst.multi INFO: cancelling 17071370 on provider checkpoint 2024-03-15 12:21:48 proc_lsst.multi:146 [INFO] found job 17071372 in provider checkpoint proc_lsst.multi INFO: found job 17071372 in provider checkpoint 2024-03-15 12:21:48 proc_lsst.multi:201 [INFO] cancelling 17071372 on provider checkpoint proc_lsst.multi INFO: cancelling 17071372 on provider checkpoint 2024-03-15 12:21:48 proc_lsst.multi:146 [INFO] found job 17071373 in provider checkpoint proc_lsst.multi INFO: found job 17071373 in provider checkpoint 2024-03-15 12:21:48 proc_lsst.multi:201 [INFO] cancelling 17071373 on provider checkpoint proc_lsst.multi INFO: cancelling 17071373 on provider checkpoint 2024-03-15 12:21:49 proc_lsst.multi:146 [INFO] found job 17071374 in provider checkpoint proc_lsst.multi INFO: found job 17071374 in provider checkpoint 2024-03-15 12:21:49 proc_lsst.multi:201 [INFO] cancelling 17071374 on provider checkpoint proc_lsst.multi INFO: cancelling 17071374 on provider checkpoint 2024-03-15 12:21:49 proc_lsst.multi:146 [INFO] found job 17071376 in provider checkpoint proc_lsst.multi INFO: found job 17071376 in provider checkpoint 2024-03-15 12:21:49 proc_lsst.multi:201 [INFO] cancelling 17071376 on provider checkpoint proc_lsst.multi INFO: cancelling 17071376 on provider checkpoint 2024-03-15 12:21:49 proc_lsst.multi:146 [INFO] found job 17071377 in provider checkpoint proc_lsst.multi INFO: found job 17071377 in provider checkpoint 2024-03-15 12:21:49 proc_lsst.multi:201 [INFO] cancelling 17071377 on provider checkpoint proc_lsst.multi INFO: cancelling 17071377 on provider checkpoint 2024-03-15 12:21:50 proc_lsst.multi:146 [INFO] found job 17071378 in provider checkpoint proc_lsst.multi INFO: found job 17071378 in provider checkpoint 2024-03-15 12:21:50 proc_lsst.multi:201 [INFO] cancelling 17071378 on provider checkpoint proc_lsst.multi INFO: cancelling 17071378 on provider checkpoint 2024-03-15 12:21:50 proc_lsst.multi:146 [INFO] found job 17071379 in provider checkpoint proc_lsst.multi INFO: found job 17071379 in provider checkpoint 2024-03-15 12:21:50 proc_lsst.multi:201 [INFO] cancelling 17071379 on provider checkpoint proc_lsst.multi INFO: cancelling 17071379 on provider checkpoint 2024-03-15 12:21:51 proc_lsst.multi:146 [INFO] found job 17071380 in provider checkpoint proc_lsst.multi INFO: found job 17071380 in provider checkpoint 2024-03-15 12:21:51 proc_lsst.multi:201 [INFO] cancelling 17071380 on provider checkpoint proc_lsst.multi INFO: cancelling 17071380 on provider checkpoint 2024-03-15 12:21:51 proc_lsst.multi:146 [INFO] found job 17071381 in provider checkpoint proc_lsst.multi INFO: found job 17071381 in provider checkpoint 2024-03-15 12:21:51 proc_lsst.multi:201 [INFO] cancelling 17071381 on provider checkpoint proc_lsst.multi INFO: cancelling 17071381 on provider checkpoint 2024-03-15 12:21:51 proc_lsst.multi:146 [INFO] found job 17071382 in provider checkpoint proc_lsst.multi INFO: found job 17071382 in provider checkpoint 2024-03-15 12:21:51 proc_lsst.multi:201 [INFO] cancelling 17071382 on provider checkpoint proc_lsst.multi INFO: cancelling 17071382 on provider checkpoint 2024-03-15 12:21:52 proc_lsst.multi:146 [INFO] found job 17071383 in provider checkpoint proc_lsst.multi INFO: found job 17071383 in provider checkpoint 2024-03-15 12:21:52 proc_lsst.multi:201 [INFO] cancelling 17071383 on provider checkpoint proc_lsst.multi INFO: cancelling 17071383 on provider checkpoint 2024-03-15 12:21:52 proc_lsst.multi:146 [INFO] found job 17071384 in provider checkpoint proc_lsst.multi INFO: found job 17071384 in provider checkpoint 2024-03-15 12:21:52 proc_lsst.multi:201 [INFO] cancelling 17071384 on provider checkpoint proc_lsst.multi INFO: cancelling 17071384 on provider checkpoint 2024-03-15 12:21:53 proc_lsst.multi:146 [INFO] found job 17071385 in provider checkpoint proc_lsst.multi INFO: found job 17071385 in provider checkpoint 2024-03-15 12:21:53 proc_lsst.multi:201 [INFO] cancelling 17071385 on provider checkpoint proc_lsst.multi INFO: cancelling 17071385 on provider checkpoint 2024-03-15 12:21:53 proc_lsst.multi:146 [INFO] found job 17071386 in provider checkpoint proc_lsst.multi INFO: found job 17071386 in provider checkpoint 2024-03-15 12:21:53 proc_lsst.multi:201 [INFO] cancelling 17071386 on provider checkpoint proc_lsst.multi INFO: cancelling 17071386 on provider checkpoint 2024-03-15 12:21:53 proc_lsst.multi:146 [INFO] found job 17071387 in provider checkpoint proc_lsst.multi INFO: found job 17071387 in provider checkpoint 2024-03-15 12:21:53 proc_lsst.multi:201 [INFO] cancelling 17071387 on provider checkpoint proc_lsst.multi INFO: cancelling 17071387 on provider checkpoint 2024-03-15 12:21:54 proc_lsst.multi:146 [INFO] found job 17071388 in provider checkpoint proc_lsst.multi INFO: found job 17071388 in provider checkpoint 2024-03-15 12:21:54 proc_lsst.multi:201 [INFO] cancelling 17071388 on provider checkpoint proc_lsst.multi INFO: cancelling 17071388 on provider checkpoint 2024-03-15 12:21:54 proc_lsst.multi:146 [INFO] found job 17071389 in provider checkpoint proc_lsst.multi INFO: found job 17071389 in provider checkpoint 2024-03-15 12:21:54 proc_lsst.multi:201 [INFO] cancelling 17071389 on provider checkpoint proc_lsst.multi INFO: cancelling 17071389 on provider checkpoint 2024-03-15 12:21:55 proc_lsst.multi:146 [INFO] found job 17071390 in provider checkpoint proc_lsst.multi INFO: found job 17071390 in provider checkpoint 2024-03-15 12:21:55 proc_lsst.multi:201 [INFO] cancelling 17071390 on provider checkpoint proc_lsst.multi INFO: cancelling 17071390 on provider checkpoint 2024-03-15 12:21:55 proc_lsst.multi:146 [INFO] found job 17071391 in provider checkpoint proc_lsst.multi INFO: found job 17071391 in provider checkpoint 2024-03-15 12:21:55 proc_lsst.multi:201 [INFO] cancelling 17071391 on provider checkpoint proc_lsst.multi INFO: cancelling 17071391 on provider checkpoint 2024-03-15 12:21:55 proc_lsst.multi:146 [INFO] found job 17071392 in provider checkpoint proc_lsst.multi INFO: found job 17071392 in provider checkpoint 2024-03-15 12:21:55 proc_lsst.multi:201 [INFO] cancelling 17071392 on provider checkpoint proc_lsst.multi INFO: cancelling 17071392 on provider checkpoint 2024-03-15 12:21:56 proc_lsst.multi:146 [INFO] found job 17071393 in provider checkpoint proc_lsst.multi INFO: found job 17071393 in provider checkpoint 2024-03-15 12:21:56 proc_lsst.multi:201 [INFO] cancelling 17071393 on provider checkpoint proc_lsst.multi INFO: cancelling 17071393 on provider checkpoint 2024-03-15 12:21:56 proc_lsst.multi:146 [INFO] found job 17071395 in provider checkpoint proc_lsst.multi INFO: found job 17071395 in provider checkpoint 2024-03-15 12:21:56 proc_lsst.multi:201 [INFO] cancelling 17071395 on provider checkpoint proc_lsst.multi INFO: cancelling 17071395 on provider checkpoint 2024-03-15 12:21:57 proc_lsst.multi:146 [INFO] found job 17071396 in provider checkpoint proc_lsst.multi INFO: found job 17071396 in provider checkpoint 2024-03-15 12:21:57 proc_lsst.multi:201 [INFO] cancelling 17071396 on provider checkpoint proc_lsst.multi INFO: cancelling 17071396 on provider checkpoint 2024-03-15 12:21:57 proc_lsst.multi:146 [INFO] found job 17071397 in provider checkpoint proc_lsst.multi INFO: found job 17071397 in provider checkpoint 2024-03-15 12:21:57 proc_lsst.multi:201 [INFO] cancelling 17071397 on provider checkpoint proc_lsst.multi INFO: cancelling 17071397 on provider checkpoint 2024-03-15 12:21:57 proc_lsst.multi:146 [INFO] found job 17071398 in provider checkpoint proc_lsst.multi INFO: found job 17071398 in provider checkpoint 2024-03-15 12:21:57 proc_lsst.multi:201 [INFO] cancelling 17071398 on provider checkpoint proc_lsst.multi INFO: cancelling 17071398 on provider checkpoint 2024-03-15 12:21:58 proc_lsst.multi:146 [INFO] found job 17071399 in provider checkpoint proc_lsst.multi INFO: found job 17071399 in provider checkpoint 2024-03-15 12:21:58 proc_lsst.multi:201 [INFO] cancelling 17071399 on provider checkpoint proc_lsst.multi INFO: cancelling 17071399 on provider checkpoint 2024-03-15 12:21:58 proc_lsst.multi:146 [INFO] found job 17071401 in provider checkpoint proc_lsst.multi INFO: found job 17071401 in provider checkpoint 2024-03-15 12:21:58 proc_lsst.multi:201 [INFO] cancelling 17071401 on provider checkpoint proc_lsst.multi INFO: cancelling 17071401 on provider checkpoint 2024-03-15 12:21:59 proc_lsst.multi:146 [INFO] found job 17071402 in provider checkpoint proc_lsst.multi INFO: found job 17071402 in provider checkpoint 2024-03-15 12:21:59 proc_lsst.multi:201 [INFO] cancelling 17071402 on provider checkpoint proc_lsst.multi INFO: cancelling 17071402 on provider checkpoint 2024-03-15 12:21:59 proc_lsst.multi:146 [INFO] found job 17071403 in provider checkpoint proc_lsst.multi INFO: found job 17071403 in provider checkpoint 2024-03-15 12:21:59 proc_lsst.multi:201 [INFO] cancelling 17071403 on provider checkpoint proc_lsst.multi INFO: cancelling 17071403 on provider checkpoint 2024-03-15 12:21:59 proc_lsst.multi:146 [INFO] found job 17071404 in provider checkpoint proc_lsst.multi INFO: found job 17071404 in provider checkpoint 2024-03-15 12:21:59 proc_lsst.multi:201 [INFO] cancelling 17071404 on provider checkpoint proc_lsst.multi INFO: cancelling 17071404 on provider checkpoint 2024-03-15 12:22:00 proc_lsst.multi:146 [INFO] found job 17071405 in provider checkpoint proc_lsst.multi INFO: found job 17071405 in provider checkpoint 2024-03-15 12:22:00 proc_lsst.multi:201 [INFO] cancelling 17071405 on provider checkpoint proc_lsst.multi INFO: cancelling 17071405 on provider checkpoint 2024-03-15 12:22:00 proc_lsst.multi:146 [INFO] found job 17071406 in provider checkpoint proc_lsst.multi INFO: found job 17071406 in provider checkpoint 2024-03-15 12:22:00 proc_lsst.multi:201 [INFO] cancelling 17071406 on provider checkpoint proc_lsst.multi INFO: cancelling 17071406 on provider checkpoint 2024-03-15 12:22:01 proc_lsst.multi:146 [INFO] found job 17071407 in provider checkpoint proc_lsst.multi INFO: found job 17071407 in provider checkpoint 2024-03-15 12:22:01 proc_lsst.multi:201 [INFO] cancelling 17071407 on provider checkpoint proc_lsst.multi INFO: cancelling 17071407 on provider checkpoint 2024-03-15 12:22:01 proc_lsst.multi:146 [INFO] found job 17071408 in provider checkpoint proc_lsst.multi INFO: found job 17071408 in provider checkpoint 2024-03-15 12:22:01 proc_lsst.multi:201 [INFO] cancelling 17071408 on provider checkpoint proc_lsst.multi INFO: cancelling 17071408 on provider checkpoint 2024-03-15 12:22:02 proc_lsst.multi:146 [INFO] found job 17071409 in provider checkpoint proc_lsst.multi INFO: found job 17071409 in provider checkpoint 2024-03-15 12:22:02 proc_lsst.multi:201 [INFO] cancelling 17071409 on provider checkpoint proc_lsst.multi INFO: cancelling 17071409 on provider checkpoint 2024-03-15 12:22:02 proc_lsst.multi:146 [INFO] found job 17071411 in provider checkpoint proc_lsst.multi INFO: found job 17071411 in provider checkpoint 2024-03-15 12:22:02 proc_lsst.multi:201 [INFO] cancelling 17071411 on provider checkpoint proc_lsst.multi INFO: cancelling 17071411 on provider checkpoint 2024-03-15 12:22:02 proc_lsst.multi:146 [INFO] found job 17071413 in provider checkpoint proc_lsst.multi INFO: found job 17071413 in provider checkpoint 2024-03-15 12:22:02 proc_lsst.multi:201 [INFO] cancelling 17071413 on provider checkpoint proc_lsst.multi INFO: cancelling 17071413 on provider checkpoint 2024-03-15 12:22:03 proc_lsst.multi:146 [INFO] found job 17071415 in provider checkpoint proc_lsst.multi INFO: found job 17071415 in provider checkpoint 2024-03-15 12:22:03 proc_lsst.multi:201 [INFO] cancelling 17071415 on provider checkpoint proc_lsst.multi INFO: cancelling 17071415 on provider checkpoint 2024-03-15 12:22:03 proc_lsst.multi:146 [INFO] found job 17071416 in provider checkpoint proc_lsst.multi INFO: found job 17071416 in provider checkpoint 2024-03-15 12:22:03 proc_lsst.multi:201 [INFO] cancelling 17071416 on provider checkpoint proc_lsst.multi INFO: cancelling 17071416 on provider checkpoint 2024-03-15 12:22:03 proc_lsst.multi:146 [INFO] found job 17071418 in provider checkpoint proc_lsst.multi INFO: found job 17071418 in provider checkpoint 2024-03-15 12:22:03 proc_lsst.multi:201 [INFO] cancelling 17071418 on provider checkpoint proc_lsst.multi INFO: cancelling 17071418 on provider checkpoint 2024-03-15 12:22:04 proc_lsst.multi:146 [INFO] found job 17071419 in provider checkpoint proc_lsst.multi INFO: found job 17071419 in provider checkpoint 2024-03-15 12:22:04 proc_lsst.multi:201 [INFO] cancelling 17071419 on provider checkpoint proc_lsst.multi INFO: cancelling 17071419 on provider checkpoint 2024-03-15 12:22:04 proc_lsst.multi:146 [INFO] found job 17071420 in provider checkpoint proc_lsst.multi INFO: found job 17071420 in provider checkpoint 2024-03-15 12:22:04 proc_lsst.multi:201 [INFO] cancelling 17071420 on provider checkpoint proc_lsst.multi INFO: cancelling 17071420 on provider checkpoint 2024-03-15 12:22:05 proc_lsst.multi:146 [INFO] found job 17071421 in provider checkpoint proc_lsst.multi INFO: found job 17071421 in provider checkpoint 2024-03-15 12:22:05 proc_lsst.multi:201 [INFO] cancelling 17071421 on provider checkpoint proc_lsst.multi INFO: cancelling 17071421 on provider checkpoint 2024-03-15 12:22:05 proc_lsst.multi:146 [INFO] found job 17071422 in provider checkpoint proc_lsst.multi INFO: found job 17071422 in provider checkpoint 2024-03-15 12:22:05 proc_lsst.multi:201 [INFO] cancelling 17071422 on provider checkpoint proc_lsst.multi INFO: cancelling 17071422 on provider checkpoint 2024-03-15 12:22:05 proc_lsst.multi:146 [INFO] found job 17071423 in provider checkpoint proc_lsst.multi INFO: found job 17071423 in provider checkpoint 2024-03-15 12:22:05 proc_lsst.multi:201 [INFO] cancelling 17071423 on provider checkpoint proc_lsst.multi INFO: cancelling 17071423 on provider checkpoint 2024-03-15 12:22:06 proc_lsst.multi:146 [INFO] found job 17071425 in provider checkpoint proc_lsst.multi INFO: found job 17071425 in provider checkpoint 2024-03-15 12:22:06 proc_lsst.multi:201 [INFO] cancelling 17071425 on provider checkpoint proc_lsst.multi INFO: cancelling 17071425 on provider checkpoint 2024-03-15 12:22:06 proc_lsst.multi:146 [INFO] found job 17071426 in provider checkpoint proc_lsst.multi INFO: found job 17071426 in provider checkpoint 2024-03-15 12:22:06 proc_lsst.multi:201 [INFO] cancelling 17071426 on provider checkpoint proc_lsst.multi INFO: cancelling 17071426 on provider checkpoint 2024-03-15 12:22:07 proc_lsst.multi:146 [INFO] found job 17071427 in provider checkpoint proc_lsst.multi INFO: found job 17071427 in provider checkpoint 2024-03-15 12:22:07 proc_lsst.multi:201 [INFO] cancelling 17071427 on provider checkpoint proc_lsst.multi INFO: cancelling 17071427 on provider checkpoint 2024-03-15 12:22:07 proc_lsst.multi:146 [INFO] found job 17071428 in provider checkpoint proc_lsst.multi INFO: found job 17071428 in provider checkpoint 2024-03-15 12:22:07 proc_lsst.multi:201 [INFO] cancelling 17071428 on provider checkpoint proc_lsst.multi INFO: cancelling 17071428 on provider checkpoint 2024-03-15 12:22:07 proc_lsst.multi:146 [INFO] found job 17071429 in provider checkpoint proc_lsst.multi INFO: found job 17071429 in provider checkpoint 2024-03-15 12:22:07 proc_lsst.multi:201 [INFO] cancelling 17071429 on provider checkpoint proc_lsst.multi INFO: cancelling 17071429 on provider checkpoint 2024-03-15 12:22:08 proc_lsst.multi:146 [INFO] found job 17071430 in provider checkpoint proc_lsst.multi INFO: found job 17071430 in provider checkpoint 2024-03-15 12:22:08 proc_lsst.multi:201 [INFO] cancelling 17071430 on provider checkpoint proc_lsst.multi INFO: cancelling 17071430 on provider checkpoint 2024-03-15 12:22:08 proc_lsst.multi:146 [INFO] found job 17071431 in provider checkpoint proc_lsst.multi INFO: found job 17071431 in provider checkpoint 2024-03-15 12:22:08 proc_lsst.multi:201 [INFO] cancelling 17071431 on provider checkpoint proc_lsst.multi INFO: cancelling 17071431 on provider checkpoint 2024-03-15 12:22:09 proc_lsst.multi:146 [INFO] found job 17071433 in provider checkpoint proc_lsst.multi INFO: found job 17071433 in provider checkpoint 2024-03-15 12:22:09 proc_lsst.multi:201 [INFO] cancelling 17071433 on provider checkpoint proc_lsst.multi INFO: cancelling 17071433 on provider checkpoint 2024-03-15 12:22:09 proc_lsst.multi:146 [INFO] found job 17071434 in provider checkpoint proc_lsst.multi INFO: found job 17071434 in provider checkpoint 2024-03-15 12:22:09 proc_lsst.multi:201 [INFO] cancelling 17071434 on provider checkpoint proc_lsst.multi INFO: cancelling 17071434 on provider checkpoint 2024-03-15 12:22:09 proc_lsst.multi:146 [INFO] found job 17071435 in provider checkpoint proc_lsst.multi INFO: found job 17071435 in provider checkpoint 2024-03-15 12:22:09 proc_lsst.multi:201 [INFO] cancelling 17071435 on provider checkpoint proc_lsst.multi INFO: cancelling 17071435 on provider checkpoint 2024-03-15 12:22:10 proc_lsst.multi:146 [INFO] found job 17071437 in provider checkpoint proc_lsst.multi INFO: found job 17071437 in provider checkpoint 2024-03-15 12:22:10 proc_lsst.multi:201 [INFO] cancelling 17071437 on provider checkpoint proc_lsst.multi INFO: cancelling 17071437 on provider checkpoint 2024-03-15 12:22:10 proc_lsst.multi:146 [INFO] found job 17071445 in provider checkpoint proc_lsst.multi INFO: found job 17071445 in provider checkpoint 2024-03-15 12:22:10 proc_lsst.multi:201 [INFO] cancelling 17071445 on provider checkpoint proc_lsst.multi INFO: cancelling 17071445 on provider checkpoint 2024-03-15 12:22:10 proc_lsst.multi:146 [INFO] found job 17071446 in provider checkpoint proc_lsst.multi INFO: found job 17071446 in provider checkpoint 2024-03-15 12:22:10 proc_lsst.multi:201 [INFO] cancelling 17071446 on provider checkpoint proc_lsst.multi INFO: cancelling 17071446 on provider checkpoint 2024-03-15 12:22:11 proc_lsst.multi:146 [INFO] found job 17071447 in provider checkpoint proc_lsst.multi INFO: found job 17071447 in provider checkpoint 2024-03-15 12:22:11 proc_lsst.multi:201 [INFO] cancelling 17071447 on provider checkpoint proc_lsst.multi INFO: cancelling 17071447 on provider checkpoint 2024-03-15 12:22:11 proc_lsst.multi:146 [INFO] found job 17071448 in provider checkpoint proc_lsst.multi INFO: found job 17071448 in provider checkpoint 2024-03-15 12:22:11 proc_lsst.multi:201 [INFO] cancelling 17071448 on provider checkpoint proc_lsst.multi INFO: cancelling 17071448 on provider checkpoint 2024-03-15 12:22:12 proc_lsst.multi:146 [INFO] found job 17071449 in provider checkpoint proc_lsst.multi INFO: found job 17071449 in provider checkpoint 2024-03-15 12:22:12 proc_lsst.multi:201 [INFO] cancelling 17071449 on provider checkpoint proc_lsst.multi INFO: cancelling 17071449 on provider checkpoint 2024-03-15 12:22:12 proc_lsst.multi:146 [INFO] found job 17071450 in provider checkpoint proc_lsst.multi INFO: found job 17071450 in provider checkpoint 2024-03-15 12:22:12 proc_lsst.multi:201 [INFO] cancelling 17071450 on provider checkpoint proc_lsst.multi INFO: cancelling 17071450 on provider checkpoint 2024-03-15 12:22:14 proc_lsst.multi:146 [INFO] found job 17071451 in provider checkpoint proc_lsst.multi INFO: found job 17071451 in provider checkpoint 2024-03-15 12:22:14 proc_lsst.multi:201 [INFO] cancelling 17071451 on provider checkpoint proc_lsst.multi INFO: cancelling 17071451 on provider checkpoint 2024-03-15 12:22:14 proc_lsst.multi:146 [INFO] found job 17071452 in provider checkpoint proc_lsst.multi INFO: found job 17071452 in provider checkpoint 2024-03-15 12:22:14 proc_lsst.multi:201 [INFO] cancelling 17071452 on provider checkpoint proc_lsst.multi INFO: cancelling 17071452 on provider checkpoint 2024-03-15 12:22:15 proc_lsst.multi:146 [INFO] found job 17071124 in provider astro proc_lsst.multi INFO: found job 17071124 in provider astro 2024-03-15 12:22:15 proc_lsst.multi:201 [INFO] cancelling 17071124 on provider astro proc_lsst.multi INFO: cancelling 17071124 on provider astro 2024-03-15 12:22:15 proc_lsst.multi:146 [INFO] found job 17071125 in provider astro proc_lsst.multi INFO: found job 17071125 in provider astro 2024-03-15 12:22:15 proc_lsst.multi:201 [INFO] cancelling 17071125 on provider astro proc_lsst.multi INFO: cancelling 17071125 on provider astro 2024-03-15 12:22:16 proc_lsst.multi:146 [INFO] found job 17071126 in provider astro proc_lsst.multi INFO: found job 17071126 in provider astro 2024-03-15 12:22:16 proc_lsst.multi:201 [INFO] cancelling 17071126 on provider astro proc_lsst.multi INFO: cancelling 17071126 on provider astro 2024-03-15 12:22:16 proc_lsst.multi:146 [INFO] found job 17071127 in provider astro proc_lsst.multi INFO: found job 17071127 in provider astro 2024-03-15 12:22:16 proc_lsst.multi:201 [INFO] cancelling 17071127 on provider astro proc_lsst.multi INFO: cancelling 17071127 on provider astro 2024-03-15 12:22:16 proc_lsst.multi:146 [INFO] found job 17071128 in provider astro proc_lsst.multi INFO: found job 17071128 in provider astro 2024-03-15 12:22:16 proc_lsst.multi:201 [INFO] cancelling 17071128 on provider astro proc_lsst.multi INFO: cancelling 17071128 on provider astro 2024-03-15 12:22:17 proc_lsst.multi:146 [INFO] found job 17071130 in provider astro proc_lsst.multi INFO: found job 17071130 in provider astro 2024-03-15 12:22:17 proc_lsst.multi:201 [INFO] cancelling 17071130 on provider astro proc_lsst.multi INFO: cancelling 17071130 on provider astro 2024-03-15 12:22:17 proc_lsst.multi:146 [INFO] found job 17071131 in provider astro proc_lsst.multi INFO: found job 17071131 in provider astro 2024-03-15 12:22:17 proc_lsst.multi:201 [INFO] cancelling 17071131 on provider astro proc_lsst.multi INFO: cancelling 17071131 on provider astro 2024-03-15 12:22:18 proc_lsst.multi:146 [INFO] found job 17071132 in provider astro proc_lsst.multi INFO: found job 17071132 in provider astro 2024-03-15 12:22:18 proc_lsst.multi:201 [INFO] cancelling 17071132 on provider astro proc_lsst.multi INFO: cancelling 17071132 on provider astro 2024-03-15 12:22:18 proc_lsst.multi:146 [INFO] found job 17071133 in provider astro proc_lsst.multi INFO: found job 17071133 in provider astro 2024-03-15 12:22:18 proc_lsst.multi:201 [INFO] cancelling 17071133 on provider astro proc_lsst.multi INFO: cancelling 17071133 on provider astro 2024-03-15 12:22:18 proc_lsst.multi:146 [INFO] found job 17071134 in provider astro proc_lsst.multi INFO: found job 17071134 in provider astro 2024-03-15 12:22:18 proc_lsst.multi:201 [INFO] cancelling 17071134 on provider astro proc_lsst.multi INFO: cancelling 17071134 on provider astro 2024-03-15 12:22:19 proc_lsst.multi:146 [INFO] found job 17071135 in provider astro proc_lsst.multi INFO: found job 17071135 in provider astro 2024-03-15 12:22:19 proc_lsst.multi:201 [INFO] cancelling 17071135 on provider astro proc_lsst.multi INFO: cancelling 17071135 on provider astro 2024-03-15 12:22:19 proc_lsst.multi:146 [INFO] found job 17071136 in provider astro proc_lsst.multi INFO: found job 17071136 in provider astro 2024-03-15 12:22:19 proc_lsst.multi:201 [INFO] cancelling 17071136 on provider astro proc_lsst.multi INFO: cancelling 17071136 on provider astro 2024-03-15 12:22:20 proc_lsst.multi:146 [INFO] found job 17071137 in provider astro proc_lsst.multi INFO: found job 17071137 in provider astro 2024-03-15 12:22:20 proc_lsst.multi:201 [INFO] cancelling 17071137 on provider astro proc_lsst.multi INFO: cancelling 17071137 on provider astro 2024-03-15 12:22:20 proc_lsst.multi:146 [INFO] found job 17071139 in provider astro proc_lsst.multi INFO: found job 17071139 in provider astro 2024-03-15 12:22:20 proc_lsst.multi:201 [INFO] cancelling 17071139 on provider astro proc_lsst.multi INFO: cancelling 17071139 on provider astro 2024-03-15 12:22:20 proc_lsst.multi:146 [INFO] found job 17071141 in provider astro proc_lsst.multi INFO: found job 17071141 in provider astro 2024-03-15 12:22:20 proc_lsst.multi:201 [INFO] cancelling 17071141 on provider astro proc_lsst.multi INFO: cancelling 17071141 on provider astro 2024-03-15 12:22:21 proc_lsst.multi:146 [INFO] found job 17071142 in provider astro proc_lsst.multi INFO: found job 17071142 in provider astro 2024-03-15 12:22:21 proc_lsst.multi:201 [INFO] cancelling 17071142 on provider astro proc_lsst.multi INFO: cancelling 17071142 on provider astro 2024-03-15 12:22:21 proc_lsst.multi:146 [INFO] found job 17071143 in provider astro proc_lsst.multi INFO: found job 17071143 in provider astro 2024-03-15 12:22:21 proc_lsst.multi:201 [INFO] cancelling 17071143 on provider astro proc_lsst.multi INFO: cancelling 17071143 on provider astro 2024-03-15 12:22:22 proc_lsst.multi:146 [INFO] found job 17071144 in provider astro proc_lsst.multi INFO: found job 17071144 in provider astro 2024-03-15 12:22:22 proc_lsst.multi:201 [INFO] cancelling 17071144 on provider astro proc_lsst.multi INFO: cancelling 17071144 on provider astro 2024-03-15 12:22:22 proc_lsst.multi:146 [INFO] found job 17071148 in provider astro proc_lsst.multi INFO: found job 17071148 in provider astro 2024-03-15 12:22:22 proc_lsst.multi:201 [INFO] cancelling 17071148 on provider astro proc_lsst.multi INFO: cancelling 17071148 on provider astro 2024-03-15 12:22:22 proc_lsst.multi:146 [INFO] found job 17071149 in provider astro proc_lsst.multi INFO: found job 17071149 in provider astro 2024-03-15 12:22:22 proc_lsst.multi:201 [INFO] cancelling 17071149 on provider astro proc_lsst.multi INFO: cancelling 17071149 on provider astro 2024-03-15 12:22:23 proc_lsst.multi:146 [INFO] found job 17071151 in provider astro proc_lsst.multi INFO: found job 17071151 in provider astro 2024-03-15 12:22:23 proc_lsst.multi:201 [INFO] cancelling 17071151 on provider astro proc_lsst.multi INFO: cancelling 17071151 on provider astro 2024-03-15 12:22:23 proc_lsst.multi:146 [INFO] found job 17071152 in provider astro proc_lsst.multi INFO: found job 17071152 in provider astro 2024-03-15 12:22:23 proc_lsst.multi:201 [INFO] cancelling 17071152 on provider astro proc_lsst.multi INFO: cancelling 17071152 on provider astro 2024-03-15 12:22:24 proc_lsst.multi:146 [INFO] found job 17071153 in provider astro proc_lsst.multi INFO: found job 17071153 in provider astro 2024-03-15 12:22:24 proc_lsst.multi:201 [INFO] cancelling 17071153 on provider astro proc_lsst.multi INFO: cancelling 17071153 on provider astro 2024-03-15 12:22:24 proc_lsst.multi:146 [INFO] found job 17071154 in provider astro proc_lsst.multi INFO: found job 17071154 in provider astro 2024-03-15 12:22:24 proc_lsst.multi:201 [INFO] cancelling 17071154 on provider astro proc_lsst.multi INFO: cancelling 17071154 on provider astro 2024-03-15 12:22:24 proc_lsst.multi:146 [INFO] found job 17071156 in provider astro proc_lsst.multi INFO: found job 17071156 in provider astro 2024-03-15 12:22:24 proc_lsst.multi:201 [INFO] cancelling 17071156 on provider astro proc_lsst.multi INFO: cancelling 17071156 on provider astro 2024-03-15 12:22:25 proc_lsst.multi:146 [INFO] found job 17071157 in provider astro proc_lsst.multi INFO: found job 17071157 in provider astro 2024-03-15 12:22:25 proc_lsst.multi:201 [INFO] cancelling 17071157 on provider astro proc_lsst.multi INFO: cancelling 17071157 on provider astro 2024-03-15 12:22:25 proc_lsst.multi:146 [INFO] found job 17071158 in provider astro proc_lsst.multi INFO: found job 17071158 in provider astro 2024-03-15 12:22:25 proc_lsst.multi:201 [INFO] cancelling 17071158 on provider astro proc_lsst.multi INFO: cancelling 17071158 on provider astro 2024-03-15 12:22:25 proc_lsst.multi:146 [INFO] found job 17071160 in provider astro proc_lsst.multi INFO: found job 17071160 in provider astro 2024-03-15 12:22:25 proc_lsst.multi:201 [INFO] cancelling 17071160 on provider astro proc_lsst.multi INFO: cancelling 17071160 on provider astro 2024-03-15 12:22:26 proc_lsst.multi:146 [INFO] found job 17071162 in provider astro proc_lsst.multi INFO: found job 17071162 in provider astro 2024-03-15 12:22:26 proc_lsst.multi:201 [INFO] cancelling 17071162 on provider astro proc_lsst.multi INFO: cancelling 17071162 on provider astro 2024-03-15 12:22:26 proc_lsst.multi:146 [INFO] found job 17071163 in provider astro proc_lsst.multi INFO: found job 17071163 in provider astro 2024-03-15 12:22:26 proc_lsst.multi:201 [INFO] cancelling 17071163 on provider astro proc_lsst.multi INFO: cancelling 17071163 on provider astro 2024-03-15 12:22:27 proc_lsst.multi:146 [INFO] found job 17071209 in provider escience_a40 proc_lsst.multi INFO: found job 17071209 in provider escience_a40 2024-03-15 12:22:27 proc_lsst.multi:201 [INFO] cancelling 17071209 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071209 on provider escience_a40 2024-03-15 12:22:27 proc_lsst.multi:146 [INFO] found job 17071210 in provider escience_a40 proc_lsst.multi INFO: found job 17071210 in provider escience_a40 2024-03-15 12:22:27 proc_lsst.multi:201 [INFO] cancelling 17071210 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071210 on provider escience_a40 2024-03-15 12:22:27 proc_lsst.multi:146 [INFO] found job 17071211 in provider escience_a40 proc_lsst.multi INFO: found job 17071211 in provider escience_a40 2024-03-15 12:22:27 proc_lsst.multi:201 [INFO] cancelling 17071211 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071211 on provider escience_a40 2024-03-15 12:22:28 proc_lsst.multi:146 [INFO] found job 17071212 in provider escience_a40 proc_lsst.multi INFO: found job 17071212 in provider escience_a40 2024-03-15 12:22:28 proc_lsst.multi:201 [INFO] cancelling 17071212 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071212 on provider escience_a40 2024-03-15 12:22:28 proc_lsst.multi:146 [INFO] found job 17071213 in provider escience_a40 proc_lsst.multi INFO: found job 17071213 in provider escience_a40 2024-03-15 12:22:28 proc_lsst.multi:201 [INFO] cancelling 17071213 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071213 on provider escience_a40 2024-03-15 12:22:29 proc_lsst.multi:146 [INFO] found job 17071214 in provider escience_a40 proc_lsst.multi INFO: found job 17071214 in provider escience_a40 2024-03-15 12:22:29 proc_lsst.multi:201 [INFO] cancelling 17071214 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071214 on provider escience_a40 2024-03-15 12:22:29 proc_lsst.multi:146 [INFO] found job 17071215 in provider escience_a40 proc_lsst.multi INFO: found job 17071215 in provider escience_a40 2024-03-15 12:22:29 proc_lsst.multi:201 [INFO] cancelling 17071215 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071215 on provider escience_a40 2024-03-15 12:22:29 proc_lsst.multi:146 [INFO] found job 17071216 in provider escience_a40 proc_lsst.multi INFO: found job 17071216 in provider escience_a40 2024-03-15 12:22:29 proc_lsst.multi:201 [INFO] cancelling 17071216 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071216 on provider escience_a40 2024-03-15 12:22:30 proc_lsst.multi:146 [INFO] found job 17071217 in provider escience_a40 proc_lsst.multi INFO: found job 17071217 in provider escience_a40 2024-03-15 12:22:30 proc_lsst.multi:201 [INFO] cancelling 17071217 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071217 on provider escience_a40 2024-03-15 12:22:30 proc_lsst.multi:146 [INFO] found job 17071218 in provider escience_a40 proc_lsst.multi INFO: found job 17071218 in provider escience_a40 2024-03-15 12:22:30 proc_lsst.multi:201 [INFO] cancelling 17071218 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071218 on provider escience_a40 2024-03-15 12:22:30 proc_lsst.multi:146 [INFO] found job 17071219 in provider escience_a40 proc_lsst.multi INFO: found job 17071219 in provider escience_a40 2024-03-15 12:22:30 proc_lsst.multi:201 [INFO] cancelling 17071219 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071219 on provider escience_a40 2024-03-15 12:22:31 proc_lsst.multi:146 [INFO] found job 17071220 in provider escience_a40 proc_lsst.multi INFO: found job 17071220 in provider escience_a40 2024-03-15 12:22:31 proc_lsst.multi:201 [INFO] cancelling 17071220 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071220 on provider escience_a40 2024-03-15 12:22:31 proc_lsst.multi:146 [INFO] found job 17071221 in provider escience_a40 proc_lsst.multi INFO: found job 17071221 in provider escience_a40 2024-03-15 12:22:31 proc_lsst.multi:201 [INFO] cancelling 17071221 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071221 on provider escience_a40 2024-03-15 12:22:32 proc_lsst.multi:146 [INFO] found job 17071222 in provider escience_a40 proc_lsst.multi INFO: found job 17071222 in provider escience_a40 2024-03-15 12:22:32 proc_lsst.multi:201 [INFO] cancelling 17071222 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071222 on provider escience_a40 2024-03-15 12:22:32 proc_lsst.multi:146 [INFO] found job 17071224 in provider escience_a40 proc_lsst.multi INFO: found job 17071224 in provider escience_a40 2024-03-15 12:22:32 proc_lsst.multi:201 [INFO] cancelling 17071224 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071224 on provider escience_a40 2024-03-15 12:22:32 proc_lsst.multi:146 [INFO] found job 17071225 in provider escience_a40 proc_lsst.multi INFO: found job 17071225 in provider escience_a40 2024-03-15 12:22:32 proc_lsst.multi:201 [INFO] cancelling 17071225 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071225 on provider escience_a40 2024-03-15 12:22:33 proc_lsst.multi:146 [INFO] found job 17071226 in provider escience_a40 proc_lsst.multi INFO: found job 17071226 in provider escience_a40 2024-03-15 12:22:33 proc_lsst.multi:201 [INFO] cancelling 17071226 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071226 on provider escience_a40 2024-03-15 12:22:33 proc_lsst.multi:146 [INFO] found job 17071227 in provider escience_a40 proc_lsst.multi INFO: found job 17071227 in provider escience_a40 2024-03-15 12:22:33 proc_lsst.multi:201 [INFO] cancelling 17071227 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071227 on provider escience_a40 2024-03-15 12:22:34 proc_lsst.multi:146 [INFO] found job 17071228 in provider escience_a40 proc_lsst.multi INFO: found job 17071228 in provider escience_a40 2024-03-15 12:22:34 proc_lsst.multi:201 [INFO] cancelling 17071228 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071228 on provider escience_a40 2024-03-15 12:22:34 proc_lsst.multi:146 [INFO] found job 17071229 in provider escience_a40 proc_lsst.multi INFO: found job 17071229 in provider escience_a40 2024-03-15 12:22:34 proc_lsst.multi:201 [INFO] cancelling 17071229 on provider escience_a40 proc_lsst.multi INFO: cancelling 17071229 on provider escience_a40 2024-03-15 12:22:34 proc_lsst.multi:146 [INFO] found job 17071164 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071164 in provider escience_rtx6k 2024-03-15 12:22:34 proc_lsst.multi:201 [INFO] cancelling 17071164 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071164 on provider escience_rtx6k 2024-03-15 12:22:34 proc_lsst.multi:146 [INFO] found job 17071165 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071165 in provider escience_rtx6k 2024-03-15 12:22:34 proc_lsst.multi:201 [INFO] cancelling 17071165 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071165 on provider escience_rtx6k 2024-03-15 12:22:35 proc_lsst.multi:146 [INFO] found job 17071166 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071166 in provider escience_rtx6k 2024-03-15 12:22:35 proc_lsst.multi:201 [INFO] cancelling 17071166 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071166 on provider escience_rtx6k 2024-03-15 12:22:35 proc_lsst.multi:146 [INFO] found job 17071199 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071199 in provider escience_rtx6k 2024-03-15 12:22:35 proc_lsst.multi:201 [INFO] cancelling 17071199 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071199 on provider escience_rtx6k 2024-03-15 12:22:35 proc_lsst.multi:146 [INFO] found job 17071208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17071208 in provider escience_rtx6k 2024-03-15 12:22:35 proc_lsst.multi:201 [INFO] cancelling 17071208 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17071208 on provider escience_rtx6k 2024-03-15 12:22:36 proc_lsst.multi:146 [INFO] found job 91474 in provider local proc_lsst.multi INFO: found job 91474 in provider local 2024-03-15 12:22:36 proc_lsst.multi:201 [INFO] cancelling 91474 on provider local proc_lsst.multi INFO: cancelling 91474 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 91474 parsl.providers.local.local WARNING: Failed to kill PID: 91474 and child processes on local 2024-03-15 12:22:37 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 378.2784 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210512_A0e_science#step0_20240315T121610Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 378.2877 seconds; current memory usage: 0.296 Gibyte, delta: 0.007 Gibyte, peak delta: 0.009 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 394.8614 seconds; current memory usage: 0.296 Gibyte, delta: 0.156 Gibyte, peak delta: 0.158 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.298 Gibyte (main), 0.298 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210512_A0e_science#step0_20240315T121610Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 6m37.340s user 1m17.381s sys 0m40.600s